/[pcre]/code/trunk/doc/pcretest.txt
ViewVC logotype

Diff of /code/trunk/doc/pcretest.txt

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 49 by nigel, Sat Feb 24 21:39:33 2007 UTC revision 63 by nigel, Sat Feb 24 21:40:03 2007 UTC
# Line 1  Line 1 
1  The pcretest program  NAME
2  --------------------       pcretest - a program  for  testing  Perl-compatible  regular
3         expressions.
4  This program is intended for testing PCRE, but it can also be used for  
5  experimenting with regular expressions.  
6    SYNOPSIS
7  If it is given two filename arguments, it reads from the first and writes to       pcretest [-d] [-i] [-m] [-o osize] [-p] [-t] [source]  [des-
8  the second. If it is given only one filename argument, it reads from that file       tination]
9  and writes to stdout. Otherwise, it reads from stdin and writes to stdout, and  
10  prompts for each line of input, using "re>" to prompt for regular expressions,       pcretest was written as a test program for the PCRE  regular
11  and "data>" to prompt for data lines.       expression  library  itself,  but  it  can  also be used for
12         experimenting  with  regular  expressions.   This   document
13  The program handles any number of sets of input on a single input file. Each       describes  the  features of the test program; for details of
14  set starts with a regular expression, and continues with any number of data       the regular  expressions  themselves,  see  the  pcrepattern
15  lines to be matched against the pattern. An empty line signals the end of the       documentation.  For details of PCRE and its options, see the
16  data lines, at which point a new regular expression is read. The regular       pcreapi documentation.
17  expressions are given enclosed in any non-alphameric delimiters other than  
18  backslash, for example  
19    OPTIONS
20    /(a|bc)x+yz/  
21    
22  White space before the initial delimiter is ignored. A regular expression may       -C        Output the version number of the PCRE library, and
23  be continued over several input lines, in which case the newline characters are                 all   available  information  about  the  optional
24  included within it. See the test input files in the testdata directory for many                 features that are included, and then exit.
25  examples. It is possible to include the delimiter within the pattern by  
26  escaping it, for example       -d        Behave as if each regex had the /D  modifier  (see
27                   below); the internal form is output after compila-
28    /abc\/def/                 tion.
29    
30  If you do so, the escape and the delimiter form part of the pattern, but since       -i        Behave as if  each  regex  had  the  /I  modifier;
31  delimiters are always non-alphameric, this does not affect its interpretation.                 information  about  the  compiled pattern is given
32  If the terminating delimiter is immediately followed by a backslash, for                 after compilation.
33  example,  
34         -m        Output the size of each compiled pattern after  it
35    /abc/\                 has been compiled. This is equivalent to adding /M
36                   to each regular expression. For compatibility with
37  then a backslash is added to the end of the pattern. This is done to provide a                 earlier  versions of pcretest, -s is a synonym for
38  way of testing the error condition that arises if a pattern finishes with a                 -m.
39  backslash, because  
40         -o osize  Set the number of elements in  the  output  vector
41    /abc\/                 that  is  used  when calling PCRE to be osize. The
42                   default value is 45, which is enough for  14  cap-
43  is interpreted as the first line of a pattern that starts with "abc/", causing                 turing  subexpressions.  The  vector  size  can be
44  pcretest to read the next line as a continuation of the regular expression.                 changed for individual matching calls by including
45                   \O in the data line (see below).
46    
47         -p        Behave as if each regex has /P modifier; the POSIX
48                   wrapper  API  is  used  to  call PCRE. None of the
49                   other options has any effect when -p is set.
50    
51         -t        Run each compile, study, and match many times with
52                   a  timer, and output resulting time per compile or
53                   match (in milliseconds). Do not set  -t  with  -m,
54                   because  you  will  then get the size output 20000
55                   times and the timing will be distorted.
56    
57    
58    DESCRIPTION
59    
60         If pcretest is given two filename arguments, it  reads  from
61         the  first and writes to the second. If it is given only one
62         filename argument, it reads from that  file  and  writes  to
63         stdout. Otherwise, it reads from stdin and writes to stdout,
64         and prompts for each line of input, using  "re>"  to  prompt
65         for  regular  expressions,  and  "data>"  to prompt for data
66         lines.
67    
68         The program handles any number of sets of input on a  single
69         input  file.  Each set starts with a regular expression, and
70         continues with any  number  of  data  lines  to  be  matched
71         against the pattern.
72    
73         Each line is matched separately and  independently.  If  you
74         want  to  do  multiple-line  matches, you have to use the \n
75         escape sequence in a single line of input to encode the new-
76         line  characters.  The maximum length of data line is 30,000
77         characters.
78    
79         An empty line signals the end of the data  lines,  at  which
80         point  a new regular expression is read. The regular expres-
81         sions are given enclosed in  any  non-alphameric  delimiters
82         other than backslash, for example
83    
84           /(a|bc)x+yz/
85    
86         White space before the initial delimiter is ignored. A regu-
87         lar expression may be continued over several input lines, in
88         which case the newline characters are included within it. It
89         is  possible  to include the delimiter within the pattern by
90         escaping it, for example
91    
92           /abc\/def/
93    
94         If you do so, the escape and the delimiter form part of  the
95         pattern,  but  since  delimiters  are always non-alphameric,
96         this does not affect its interpretation.  If the terminating
97         delimiter  is immediately followed by a backslash, for exam-
98         ple,
99    
100           /abc/\
101    
102         then a backslash is added to the end of the pattern. This is
103         done  to  provide  a way of testing the error condition that
104         arises if a pattern finishes with a backslash, because
105    
106           /abc\/
107    
108         is interpreted as the first line of a  pattern  that  starts
109         with  "abc/",  causing  pcretest  to read the next line as a
110         continuation of the regular expression.
111    
112    
113  PATTERN MODIFIERS  PATTERN MODIFIERS
 -----------------  
114    
115  The pattern may be followed by i, m, s, or x to set the PCRE_CASELESS,       The pattern may be followed by i, m, s,  or  x  to  set  the
116  PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED options, respectively. For       PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED
117  example:       options, respectively. For example:
118    
119    /caseless/i         /caseless/i
120    
121  These modifier letters have the same effect as they do in Perl. There are       These modifier letters have the same effect as  they  do  in
122  others which set PCRE options that do not correspond to anything in Perl: /A,       Perl.  There  are  others which set PCRE options that do not
123  /E, and /X set PCRE_ANCHORED, PCRE_DOLLAR_ENDONLY, and PCRE_EXTRA respectively.       correspond  to  anything  in  Perl:   /A,  /E,  and  /X  set
124         PCRE_ANCHORED,  PCRE_DOLLAR_ENDONLY,  and PCRE_EXTRA respec-
125  Searching for all possible matches within each subject string can be requested       tively.
126  by the /g or /G modifier. After finding a match, PCRE is called again to search  
127  the remainder of the subject string. The difference between /g and /G is that       Searching for  all  possible  matches  within  each  subject
128  the former uses the startoffset argument to pcre_exec() to start searching at       string  can  be  requested  by  the /g or /G modifier. After
129  a new point within the entire string (which is in effect what Perl does),       finding  a  match,  PCRE  is  called  again  to  search  the
130  whereas the latter passes over a shortened substring. This makes a difference       remainder  of  the subject string. The difference between /g
131  to the matching process if the pattern begins with a lookbehind assertion       and /G is that the former uses the startoffset  argument  to
132  (including \b or \B).       pcre_exec()  to  start  searching  at a new point within the
133         entire string (which is in effect what Perl  does),  whereas
134  If any call to pcre_exec() in a /g or /G sequence matches an empty string, the       the  latter  passes over a shortened substring. This makes a
135  next call is done with the PCRE_NOTEMPTY and PCRE_ANCHORED flags set in order       difference to the matching process  if  the  pattern  begins
136  to search for another, non-empty, match at the same point. If this second match       with a lookbehind assertion (including \b or \B).
137  fails, the start offset is advanced by one, and the normal match is retried.  
138  This imitates the way Perl handles such cases when using the /g modifier or the       If any call to pcre_exec() in a /g or /G sequence matches an
139  split() function.       empty  string,  the next call is done with the PCRE_NOTEMPTY
140         and PCRE_ANCHORED flags set in order to search for  another,
141  There are a number of other modifiers for controlling the way pcretest       non-empty,  match  at  the same point.  If this second match
142  operates.       fails, the start offset is advanced by one, and  the  normal
143         match  is  retried.  This imitates the way Perl handles such
144  The /+ modifier requests that as well as outputting the substring that matched       cases when using the /g modifier or the split() function.
145  the entire pattern, pcretest should in addition output the remainder of the  
146  subject string. This is useful for tests where the subject contains multiple       There are a number of other modifiers  for  controlling  the
147  copies of the same substring.       way pcretest operates.
148    
149  The /L modifier must be followed directly by the name of a locale, for example,       The /+ modifier requests that as well as outputting the sub-
150         string  that  matched the entire pattern, pcretest should in
151    /pattern/Lfr       addition output the remainder of the subject string. This is
152         useful  for tests where the subject contains multiple copies
153  For this reason, it must be the last modifier letter. The given locale is set,       of the same substring.
154  pcre_maketables() is called to build a set of character tables for the locale,  
155  and this is then passed to pcre_compile() when compiling the regular       The /L modifier must be followed directly by the name  of  a
156  expression. Without an /L modifier, NULL is passed as the tables pointer; that       locale, for example,
157  is, /L applies only to the expression on which it appears.  
158           /pattern/Lfr
159  The /I modifier requests that pcretest output information about the compiled  
160  expression (whether it is anchored, has a fixed first character, and so on). It       For this reason, it must be the last  modifier  letter.  The
161  does this by calling pcre_fullinfo() after compiling an expression, and       given  locale is set, pcre_maketables() is called to build a
162  outputting the information it gets back. If the pattern is studied, the results       set of character tables for the locale,  and  this  is  then
163  of that are also output.       passed  to pcre_compile() when compiling the regular expres-
164         sion. Without an /L modifier, NULL is passed as  the  tables
165  The /D modifier is a PCRE debugging feature, which also assumes /I. It causes       pointer; that is, /L applies only to the expression on which
166  the internal form of compiled regular expressions to be output after       it appears.
167  compilation.  
168         The /I modifier requests that  pcretest  output  information
169  The /S modifier causes pcre_study() to be called after the expression has been       about the compiled expression (whether it is anchored, has a
170  compiled, and the results used when the expression is matched.       fixed first character, and so on). It does this  by  calling
171         pcre_fullinfo()  after  compiling an expression, and output-
172  The /M modifier causes the size of memory block used to hold the compiled       ting the information it gets back. If the  pattern  is  stu-
173  pattern to be output.       died, the results of that are also output.
174    
175  The /P modifier causes pcretest to call PCRE via the POSIX wrapper API rather       The /D modifier is a  PCRE  debugging  feature,  which  also
176  than its native API. When this is done, all other modifiers except /i, /m, and       assumes /I.  It causes the internal form of compiled regular
177  /+ are ignored. REG_ICASE is set if /i is present, and REG_NEWLINE is set if /m       expressions to be output after compilation. If  the  pattern
178  is present. The wrapper functions force PCRE_DOLLAR_ENDONLY always, and       was studied, the information returned is also output.
179  PCRE_DOTALL unless REG_NEWLINE is set.  
180         The /S modifier causes pcre_study() to be called  after  the
181  The /8 modifier causes pcretest to call PCRE with the PCRE_UTF8 option set.       expression  has been compiled, and the results used when the
182  This turns on the (currently incomplete) support for UTF-8 character handling       expression is matched.
183  in PCRE, provided that it was compiled with this support enabled. This modifier  
184  also causes any non-printing characters in output strings to be printed using       The /M modifier causes the size of memory block used to hold
185  the \x{hh...} notation if they are valid UTF-8 sequences.       the compiled pattern to be output.
186    
187         The /P modifier causes pcretest to call PCRE via  the  POSIX
188         wrapper  API  rather than its native API. When this is done,
189         all other modifiers except  /i,  /m,  and  /+  are  ignored.
190         REG_ICASE is set if /i is present, and REG_NEWLINE is set if
191         /m    is    present.    The    wrapper    functions    force
192         PCRE_DOLLAR_ENDONLY    always,    and   PCRE_DOTALL   unless
193         REG_NEWLINE is set.
194    
195         The /8 modifier  causes  pcretest  to  call  PCRE  with  the
196         PCRE_UTF8  option set. This turns on support for UTF-8 char-
197         acter handling in PCRE, provided that it was  compiled  with
198         this  support  enabled.  This  modifier also causes any non-
199         printing characters in output strings to  be  printed  using
200         the \x{hh...} notation if they are valid UTF-8 sequences.
201    
202    
203    CALLOUTS
204    
205         If the pattern contains  any  callout  requests,  pcretest's
206         callout function will be called. By default, it displays the
207         callout number, and the start and current positions  in  the
208         text at the callout time. For example, the output
209    
210           --->pqrabcdef
211             0    ^  ^
212    
213         indicates that callout number 0 occurred for a match attempt
214         starting at the fourth character of the subject string, when
215         the pointer was at the seventh character. The callout  func-
216         tion returns zero (carry on matching) by default.
217    
218         Inserting callouts may be helpful  when  using  pcretest  to
219         check  complicated regular expressions. For further informa-
220         tion about callouts, see the pcrecallout documentation.
221    
222         For testing the PCRE library, additional control of  callout
223         behaviour  is available via escape sequences in the data, as
224         described in the following section.  In  particular,  it  is
225         possible to pass in a number as callout data (the default is
226         zero). If the callout function receives a  non-zero  number,
227         it returns that value instead of zero.
228    
229    
230  DATA LINES  DATA LINES
 ----------  
   
 Before each data line is passed to pcre_exec(), leading and trailing whitespace  
 is removed, and it is then scanned for \ escapes. The following are recognized:  
231    
232    \a         alarm (= BEL)       Before each data line is passed to pcre_exec(), leading  and
233    \b         backspace       trailing whitespace is removed, and it is then scanned for \
234    \e         escape       escapes.  Some  of  these  are  pretty  esoteric   features,
235    \f         formfeed       intended  for  checking  out  some  of  the more complicated
236    \n         newline       features of PCRE. If you are just testing "ordinary" regular
237    \r         carriage return       expressions,  you probably don't need any of these. The fol-
238    \t         tab       lowing escapes are recognized:
239    \v         vertical tab  
240    \nnn       octal character (up to 3 octal digits)         \a         alarm (= BEL)
241    \xhh       hexadecimal character (up to 2 hex digits)         \b         backspace
242    \x{hh...}  hexadecimal UTF-8 character         \e         escape
243           \f         formfeed
244    \A         pass the PCRE_ANCHORED option to pcre_exec()         \n         newline
245    \B         pass the PCRE_NOTBOL option to pcre_exec()         \r         carriage return
246    \Cdd       call pcre_copy_substring() for substring dd after a successful         \t         tab
247                 match (any decimal number less than 32)         \v         vertical tab
248    \Gdd       call pcre_get_substring() for substring dd after a successful         \nnn       octal character (up to 3 octal digits)
249                 match (any decimal number less than 32)         \xhh       hexadecimal character (up to 2 hex digits)
250    \L         call pcre_get_substringlist() after a successful match         \x{hh...}  hexadecimal character, any number of digits
251    \N         pass the PCRE_NOTEMPTY option to pcre_exec()                      in UTF-8 mode
252    \Odd       set the size of the output vector passed to pcre_exec() to dd         \A         pass the PCRE_ANCHORED option to pcre_exec()
253                 (any number of decimal digits)         \B         pass the PCRE_NOTBOL option to pcre_exec()
254    \Z         pass the PCRE_NOTEOL option to pcre_exec()         \Cdd       call pcre_copy_substring() for substring dd
255                        after a successful match (any decimal number
256  A backslash followed by anything else just escapes the anything else. If the                      less than 32)
257  very last character is a backslash, it is ignored. This gives a way of passing         \Cname     call pcre_copy_named_substring() for substring
258  an empty line as data, since a real empty line terminates the data input.                      "name" after a successful match (name termin-
259                        ated by next non alphanumeric character)
260  If /P was present on the regex, causing the POSIX wrapper API to be used, only         \C+        show the current captured substrings at callout
261  \B, and \Z have any effect, causing REG_NOTBOL and REG_NOTEOL to be passed to                      time
262  regexec() respectively.  
263           C-        do not supply a callout function
264  The use of \x{hh...} to represent UTF-8 characters is not dependent on the use         \C!n       return 1 instead of 0 when callout number n is
265  of the /8 modifier on the pattern. It is recognized always. There may be any                      reached
266  number of hexadecimal digits inside the braces. The result is from one to six         \C!n!m     return 1 instead of 0 when callout number n is
267  bytes, encoded according to the UTF-8 rules.                      reached for the nth time
268           \C*n       pass the number n (may be negative) as callout
269                        data
270           \Gdd       call pcre_get_substring() for substring dd
271                        after a successful match (any decimal number
272                        less than 32)
273           \Gname     call pcre_get_named_substring() for substring
274                        "name" after a successful match (name termin-
275                        ated by next non-alphanumeric character)
276           \L         call pcre_get_substringlist() after a
277                        successful match
278           \M         discover the minimum MATCH_LIMIT setting
279           \N         pass the PCRE_NOTEMPTY option to pcre_exec()
280           \Odd       set the size of the output vector passed to
281                        pcre_exec() to dd (any number of decimal
282                        digits)
283           \Z         pass the PCRE_NOTEOL option to pcre_exec()
284    
285         If \M is present, pcretest calls pcre_exec() several  times,
286         with  different  values  in  the  match_limit  field  of the
287         pcre_extra data structure, until it finds the minimum number
288         that is needed for pcre_exec() to complete. This number is a
289         measure of the amount of  recursion  and  backtracking  that
290         takes  place,  and  checking  it out can be instructive. For
291         most simple matches, the number is quite small, but for pat-
292         terns  with very large numbers of matching possibilities, it
293         can become large very quickly with increasing length of sub-
294         ject string.
295    
296         When \O is used, it may be higher or lower than the size set
297         by  the  -O  option (or defaulted to 45); \O applies only to
298         the call of pcre_exec() for the line in which it appears.
299    
300         A backslash followed by anything else just escapes the  any-
301         thing else. If the very last character is a backslash, it is
302         ignored. This gives a way of passing an empty line as  data,
303         since a real empty line terminates the data input.
304    
305         If /P was present on the regex, causing  the  POSIX  wrapper
306         API  to  be  used,  only  B,  and Z have any effect, causing
307         REG_NOTBOL and REG_NOTEOL to be passed to regexec()  respec-
308         tively.
309    
310         The use of \x{hh...} to represent UTF-8  characters  is  not
311         dependent  on  the use of the /8 modifier on the pattern. It
312         is recognized always. There may be any number of hexadecimal
313         digits  inside  the  braces.  The  result is from one to six
314         bytes, encoded according to the UTF-8 rules.
315    
316    
317  OUTPUT FROM PCRETEST  OUTPUT FROM PCRETEST
 --------------------  
318    
319  When a match succeeds, pcretest outputs the list of captured substrings that       When a match succeeds, pcretest outputs the list of captured
320  pcre_exec() returns, starting with number 0 for the string that matched the       substrings  that pcre_exec() returns, starting with number 0
321  whole pattern. Here is an example of an interactive pcretest run.       for the string that matched the whole pattern.  Here  is  an
322         example of an interactive pcretest run.
323    $ pcretest  
324    PCRE version 2.06 08-Jun-1999         $ pcretest
325           PCRE version 4.00 08-Jan-2003
326      re> /^abc(\d+)/  
327    data> abc123           re> /^abc(\d+)/
328     0: abc123         data> abc123
329     1: 123          0: abc123
330    data> xyz          1: 123
331    No match         data> xyz
332           No match
333  If the strings contain any non-printing characters, they are output as \0x  
334  escapes, or as \x{...} escapes if the /8 modifier was present on the pattern.       If the strings contain any non-printing characters, they are
335  If the pattern has the /+ modifier, then the output for substring 0 is followed       output  as  \0x  escapes,  or  as  \x{...} escapes if the /8
336  by the the rest of the subject string, identified by "0+" like this:       modifier was present on the pattern. If the pattern has  the
337         /+  modifier, then the output for substring 0 is followed by
338      re> /cat/+       the the rest of the subject string, identified by "0+"  like
339    data> cataract       this:
340     0: cat  
341     0+ aract           re> /cat/+
342           data> cataract
343  If the pattern has the /g or /G modifier, the results of successive matching          0: cat
344  attempts are output in sequence, like this:          0+ aract
345    
346      re> /\Bi(\w\w)/g       If the pattern has the /g or /G  modifier,  the  results  of
347    data> Mississippi       successive  matching  attempts  are output in sequence, like
348     0: iss       this:
349     1: ss  
350     0: iss           re> /\Bi(\w\w)/g
351     1: ss         data> Mississippi
352     0: ipp          0: iss
353     1: pp          1: ss
354            0: iss
355  "No match" is output only if the first match attempt fails.          1: ss
356            0: ipp
357  If any of \C, \G, or \L are present in a data line that is successfully          1: pp
358  matched, the substrings extracted by the convenience functions are output with  
359  C, G, or L after the string number instead of a colon. This is in addition to       "No match" is output only if the first match attempt fails.
360  the normal full list. The string length (that is, the return from the  
361  extraction function) is given in parentheses after each string for \C and \G.       If any of the sequences \C, \G, or \L are present in a  data
362         line  that is successfully matched, the substrings extracted
363  Note that while patterns can be continued over several lines (a plain ">"       by the convenience functions are output  with  C,  G,  or  L
364  prompt is used for continuations), data lines may not. However newlines can be       after the string number instead of a colon. This is in addi-
365  included in data by means of the \n escape.       tion to the normal full list. The string  length  (that  is,
366         the  return  from  the  extraction  function)  is  given  in
367         parentheses after each string for \C and \G.
368  COMMAND LINE OPTIONS  
369  --------------------       Note that while patterns can be continued over several lines
370         (a  plain  ">" prompt is used for continuations), data lines
371  If the -p option is given to pcretest, it is equivalent to adding /P to each       may not. However newlines can be included in data  by  means
372  regular expression: the POSIX wrapper API is used to call PCRE. None of the       of the \n escape.
373  following flags has any effect in this case.  
374    
375  If the option -d is given to pcretest, it is equivalent to adding /D to each  AUTHOR
376  regular expression: the internal form is output after compilation.  
377         Philip Hazel <ph10@cam.ac.uk>
378  If the option -i is given to pcretest, it is equivalent to adding /I to each       University Computing Service,
379  regular expression: information about the compiled pattern is given after       Cambridge CB2 3QG, England.
 compilation.  
   
 If the option -m is given to pcretest, it outputs the size of each compiled  
 pattern after it has been compiled. It is equivalent to adding /M to each  
 regular expression. For compatibility with earlier versions of pcretest, -s is  
 a synonym for -m.  
   
 If the -t option is given, each compile, study, and match is run 20000 times  
 while being timed, and the resulting time per compile or match is output in  
 milliseconds. Do not set -t with -m, because you will then get the size output  
 20000 times and the timing will be distorted. If you want to change the number  
 of repetitions used for timing, edit the definition of LOOPREPEAT at the top of  
 pcretest.c  
380    
381  Philip Hazel <ph10@cam.ac.uk>  Last updated: 03 February 2003
382  August 2000  Copyright (c) 1997-2003 University of Cambridge.
Legend:
Removed from v.49  
changed lines
  Added in v.63
webmaster@exim.org
 ViewVC Help
Powered by ViewVC 1.1.12