trunk/doc/pcrepartial.3

.TH PCREPARTIAL 3
.SH NAME
PCRE - Perl-compatible regular expressions
.SH "PARTIAL MATCHING IN PCRE"
.rs
.sp
In normal use of PCRE, if the subject string that is passed to
\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP matches as far as it goes, but is
too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There
are circumstances where it might be helpful to distinguish this case from other
cases in which there is no match.
.P
Consider, for example, an application where a human is required to type in data
for a field with specific formatting requirements. An example might be a date
in the form \fIddmmmyy\fP, defined by this pattern:
.sp
  ^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$
.sp
If the application sees the user's keystrokes one by one, and can check that
what has been typed so far is potentially valid, it is able to raise an error
as soon as a mistake is made, possibly beeping and not reflecting the
character that has been typed. This immediate feedback is likely to be a better
user interface than a check that is delayed until the entire string has been
entered.
.P
PCRE supports the concept of partial matching by means of the PCRE_PARTIAL
option, which can be set when calling \fBpcre_exec()\fP or
\fBpcre_dfa_exec()\fP. When this flag is set for \fBpcre_exec()\fP, the return
code PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time
during the matching process the last part of the subject string matched part of
the pattern. Unfortunately, for non-anchored matching, it is not possible to
obtain the position of the start of the partial match. No captured data is set
when PCRE_ERROR_PARTIAL is returned.
.P
When PCRE_PARTIAL is set for \fBpcre_dfa_exec()\fP, the return code
PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of the
subject is reached, there have been no complete matches, but there is still at
least one matching possibility. The portion of the string that provided the
partial match is set as the first matching string.
.P
Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers the
last literal byte in a pattern, and abandons matching immediately if such a
byte is not present in the subject string. This optimization cannot be used
for a subject string that might match only partially.
.
.
.SH "RESTRICTED PATTERNS FOR PCRE_PARTIAL"
.rs
.sp
Because of the way certain internal optimizations are implemented in the
\fBpcre_exec()\fP function, the PCRE_PARTIAL option cannot be used with all
patterns. These restrictions do not apply when \fBpcre_dfa_exec()\fP is used.
For \fBpcre_exec()\fP, repeated single characters such as
.sp
  a{2,4}
.sp
and repeated single metasequences such as
.sp
  \ed+
.sp
are not permitted if the maximum number of occurrences is greater than one.
Optional items such as \ed? (where the maximum is one) are permitted.
Quantifiers with any values are permitted after parentheses, so the invalid
examples above can be coded thus:
.sp
  (a){2,4}
  (\ed)+
.sp
These constructions run more slowly, but for the kinds of application that are
envisaged for this facility, this is not felt to be a major restriction.
.P
If PCRE_PARTIAL is set for a pattern that does not conform to the restrictions,
\fBpcre_exec()\fP returns the error code PCRE_ERROR_BADPARTIAL (-13).
.
.
.SH "EXAMPLE OF PARTIAL MATCHING USING PCRETEST"
.rs
.sp
If the escape sequence \eP is present in a \fBpcretest\fP data line, the
PCRE_PARTIAL flag is used for the match. Here is a run of \fBpcretest\fP that
uses the date example quoted above:
.sp
    re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
  data> 25jun04\eP
   0: 25jun04
   1: jun
  data> 25dec3\eP
  Partial match
  data> 3ju\eP
  Partial match
  data> 3juj\eP
  No match
  data> j\eP
  No match
.sp
The first data string is matched completely, so \fBpcretest\fP shows the
matched substrings. The remaining four strings do not match the complete
pattern, but the first two are partial matches. The same test, using
\fBpcre_dfa_exec()\fP matching (by means of the \eD escape sequence), produces
the following output:
.sp
    re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
  data> 25jun04\eP\eD
   0: 25jun04
  data> 23dec3\eP\eD
  Partial match: 23dec3
  data> 3ju\eP\eD
  Partial match: 3ju
  data> 3juj\eP\eD
  No match
  data> j\eP\eD
  No match
.sp
Notice that in this case the portion of the string that was matched is made
available.
.
.
.SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()"
.rs
.sp
When a partial match has been found using \fBpcre_dfa_exec()\fP, it is possible
to continue the match by providing additional subject data and calling
\fBpcre_dfa_exec()\fP again with the same compiled regular expression, this
time setting the PCRE_DFA_RESTART option. You must also pass the same working
space as before, because this is where details of the previous partial match
are stored. Here is an example using \fBpcretest\fP, using the \eR escape
sequence to set the PCRE_DFA_RESTART option (\eP and \eD are as above):
.sp
    re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
  data> 23ja\eP\eD
  Partial match: 23ja
  data> n05\eR\eD
   0: n05
.sp
The first call has "23ja" as the subject, and requests partial matching; the
second call has "n05" as the subject for the continued (restarted) match.
Notice that when the match is complete, only the last part is shown; PCRE does
not retain the previously partially-matched string. It is up to the calling
program to do that if it needs to.
.P
You can set PCRE_PARTIAL with PCRE_DFA_RESTART to continue partial matching
over multiple segments. This facility can be used to pass very long subject
strings to \fBpcre_dfa_exec()\fP. However, some care is needed for certain
types of pattern.
.P
1. If the pattern contains tests for the beginning or end of a line, you need
to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropriate, when the
subject string for any call does not contain the beginning or end of a line.
.P
2. If the pattern contains backward assertions (including \eb or \eB), you need
to arrange for some overlap in the subject strings to allow for this. For
example, you could pass the subject in chunks that are 500 bytes long, but in
a buffer of 700 bytes, with the starting offset set to 200 and the previous 200
bytes at the start of the buffer.
.P
3. Matching a subject string that is split into multiple segments does not
always produce exactly the same result as matching over one single long string.
The difference arises when there are multiple matching possibilities, because a
partial match result is given only when there are no completed matches in a
call to fBpcre_dfa_exec()\fP. This means that as soon as the shortest match has
been found, continuation to a new subject segment is no longer possible.
Consider this \fBpcretest\fP example:
.sp
    re> /dog(sbody)?/
  data> do\eP\eD
  Partial match: do
  data> gsb\eR\eP\eD
   0: g
  data> dogsbody\eD
   0: dogsbody
   1: dog
.sp
The pattern matches the words "dog" or "dogsbody". When the subject is
presented in several parts ("do" and "gsb" being the first two) the match stops
when "dog" has been found, and it is not possible to continue. On the other
hand, if "dogsbody" is presented as a single string, both matches are found.
.P
Because of this phenomenon, it does not usually make sense to end a pattern
that is going to be matched in this way with a variable repeat.
.P
4. Patterns that contain alternatives at the top level which do not all
start with the same pattern item may not work as expected. For example,
consider this pattern:
.sp
  1234|3789
.sp
If the first part of the subject is "ABC123", a partial match of the first
alternative is found at offset 3. There is no partial match for the second
alternative, because such a match does not start at the same point in the
subject string. Attempting to continue with the string "789" does not yield a
match because only those alternatives that match at one point in the subject
are remembered. The problem arises because the start of the second alternative
matches within the first alternative. There is no problem with anchored
patterns or patterns such as:
.sp
  1234|ABCD
.sp
where no string can be a partial match for both alternatives.
.
.
.SH AUTHOR
.rs
.sp
.nf
Philip Hazel
University Computing Service
Cambridge CB2 3QH, England.
.fi
.
.
.SH REVISION
.rs
.sp
.nf
Last updated: 06 March 2007
Copyright (c) 1997-2007 University of Cambridge.
.fi
1	nigel	79	.TH PCREPARTIAL 3
2	nigel	75	.SH NAME
3			PCRE - Perl-compatible regular expressions
4			.SH "PARTIAL MATCHING IN PCRE"
5			.rs
6			.sp
7			In normal use of PCRE, if the subject string that is passed to
8	nigel	77	\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP matches as far as it goes, but is
9			too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There
10			are circumstances where it might be helpful to distinguish this case from other
11			cases in which there is no match.
12	nigel	75	.P
13			Consider, for example, an application where a human is required to type in data
14			for a field with specific formatting requirements. An example might be a date
15			in the form \fIddmmmyy\fP, defined by this pattern:
16			.sp
17			^\ed?\ed(jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\ed\ed$
18			.sp
19			If the application sees the user's keystrokes one by one, and can check that
20			what has been typed so far is potentially valid, it is able to raise an error
21			as soon as a mistake is made, possibly beeping and not reflecting the
22			character that has been typed. This immediate feedback is likely to be a better
23			user interface than a check that is delayed until the entire string has been
24			entered.
25			.P
26			PCRE supports the concept of partial matching by means of the PCRE_PARTIAL
27	nigel	77	option, which can be set when calling \fBpcre_exec()\fP or
28			\fBpcre_dfa_exec()\fP. When this flag is set for \fBpcre_exec()\fP, the return
29			code PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if at any time
30			during the matching process the last part of the subject string matched part of
31			the pattern. Unfortunately, for non-anchored matching, it is not possible to
32			obtain the position of the start of the partial match. No captured data is set
33			when PCRE_ERROR_PARTIAL is returned.
34	nigel	75	.P
35	nigel	77	When PCRE_PARTIAL is set for \fBpcre_dfa_exec()\fP, the return code
36			PCRE_ERROR_NOMATCH is converted into PCRE_ERROR_PARTIAL if the end of the
37			subject is reached, there have been no complete matches, but there is still at
38			least one matching possibility. The portion of the string that provided the
39			partial match is set as the first matching string.
40			.P
41	nigel	75	Using PCRE_PARTIAL disables one of PCRE's optimizations. PCRE remembers the
42			last literal byte in a pattern, and abandons matching immediately if such a
43			byte is not present in the subject string. This optimization cannot be used
44			for a subject string that might match only partially.
45			.
46			.
47			.SH "RESTRICTED PATTERNS FOR PCRE_PARTIAL"
48			.rs
49			.sp
50	nigel	77	Because of the way certain internal optimizations are implemented in the
51			\fBpcre_exec()\fP function, the PCRE_PARTIAL option cannot be used with all
52			patterns. These restrictions do not apply when \fBpcre_dfa_exec()\fP is used.
53			For \fBpcre_exec()\fP, repeated single characters such as
54	nigel	75	.sp
55			a{2,4}
56			.sp
57			and repeated single metasequences such as
58			.sp
59			\ed+
60			.sp
61			are not permitted if the maximum number of occurrences is greater than one.
62			Optional items such as \ed? (where the maximum is one) are permitted.
63			Quantifiers with any values are permitted after parentheses, so the invalid
64			examples above can be coded thus:
65			.sp
66			(a){2,4}
67			(\ed)+
68			.sp
69			These constructions run more slowly, but for the kinds of application that are
70			envisaged for this facility, this is not felt to be a major restriction.
71			.P
72			If PCRE_PARTIAL is set for a pattern that does not conform to the restrictions,
73			\fBpcre_exec()\fP returns the error code PCRE_ERROR_BADPARTIAL (-13).
74			.
75			.
76			.SH "EXAMPLE OF PARTIAL MATCHING USING PCRETEST"
77			.rs
78			.sp
79			If the escape sequence \eP is present in a \fBpcretest\fP data line, the
80			PCRE_PARTIAL flag is used for the match. Here is a run of \fBpcretest\fP that
81			uses the date example quoted above:
82			.sp
83			re> /^\ed?\ed(jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\ed\ed$/
84	nigel	87	data> 25jun04\eP
85	nigel	75	0: 25jun04
86			1: jun
87	nigel	87	data> 25dec3\eP
88	nigel	75	Partial match
89	nigel	87	data> 3ju\eP
90	nigel	75	Partial match
91	nigel	87	data> 3juj\eP
92	nigel	75	No match
93	nigel	87	data> j\eP
94	nigel	75	No match
95			.sp
96			The first data string is matched completely, so \fBpcretest\fP shows the
97			matched substrings. The remaining four strings do not match the complete
98	nigel	93	pattern, but the first two are partial matches. The same test, using
99			\fBpcre_dfa_exec()\fP matching (by means of the \eD escape sequence), produces
100			the following output:
101	nigel	77	.sp
102			re> /^\d?\d(jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\d\d$/
103			data> 25jun04\eP\eD
104			0: 25jun04
105			data> 23dec3\eP\eD
106			Partial match: 23dec3
107			data> 3ju\eP\eD
108			Partial match: 3ju
109			data> 3juj\eP\eD
110			No match
111			data> j\eP\eD
112			No match
113			.sp
114			Notice that in this case the portion of the string that was matched is made
115			available.
116	nigel	75	.
117			.
118	nigel	77	.SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()"
119			.rs
120			.sp
121			When a partial match has been found using \fBpcre_dfa_exec()\fP, it is possible
122			to continue the match by providing additional subject data and calling
123	nigel	93	\fBpcre_dfa_exec()\fP again with the same compiled regular expression, this
124			time setting the PCRE_DFA_RESTART option. You must also pass the same working
125			space as before, because this is where details of the previous partial match
126			are stored. Here is an example using \fBpcretest\fP, using the \eR escape
127			sequence to set the PCRE_DFA_RESTART option (\eP and \eD are as above):
128	nigel	77	.sp
129			re> /^\d?\d(jan\|feb\|mar\|apr\|may\|jun\|jul\|aug\|sep\|oct\|nov\|dec)\d\d$/
130			data> 23ja\eP\eD
131			Partial match: 23ja
132			data> n05\eR\eD
133			0: n05
134			.sp
135			The first call has "23ja" as the subject, and requests partial matching; the
136			second call has "n05" as the subject for the continued (restarted) match.
137			Notice that when the match is complete, only the last part is shown; PCRE does
138			not retain the previously partially-matched string. It is up to the calling
139			program to do that if it needs to.
140	nigel	75	.P
141	nigel	93	You can set PCRE_PARTIAL with PCRE_DFA_RESTART to continue partial matching
142			over multiple segments. This facility can be used to pass very long subject
143			strings to \fBpcre_dfa_exec()\fP. However, some care is needed for certain
144			types of pattern.
145	nigel	77	.P
146			1. If the pattern contains tests for the beginning or end of a line, you need
147			to pass the PCRE_NOTBOL or PCRE_NOTEOL options, as appropriate, when the
148			subject string for any call does not contain the beginning or end of a line.
149			.P
150			2. If the pattern contains backward assertions (including \eb or \eB), you need
151			to arrange for some overlap in the subject strings to allow for this. For
152	nigel	93	example, you could pass the subject in chunks that are 500 bytes long, but in
153	nigel	77	a buffer of 700 bytes, with the starting offset set to 200 and the previous 200
154			bytes at the start of the buffer.
155			.P
156			3. Matching a subject string that is split into multiple segments does not
157			always produce exactly the same result as matching over one single long string.
158			The difference arises when there are multiple matching possibilities, because a
159			partial match result is given only when there are no completed matches in a
160			call to fBpcre_dfa_exec()\fP. This means that as soon as the shortest match has
161			been found, continuation to a new subject segment is no longer possible.
162			Consider this \fBpcretest\fP example:
163			.sp
164			re> /dog(sbody)?/
165			data> do\eP\eD
166			Partial match: do
167			data> gsb\eR\eP\eD
168			0: g
169			data> dogsbody\eD
170			0: dogsbody
171			1: dog
172			.sp
173			The pattern matches the words "dog" or "dogsbody". When the subject is
174			presented in several parts ("do" and "gsb" being the first two) the match stops
175			when "dog" has been found, and it is not possible to continue. On the other
176			hand, if "dogsbody" is presented as a single string, both matches are found.
177			.P
178			Because of this phenomenon, it does not usually make sense to end a pattern
179			that is going to be matched in this way with a variable repeat.
180	nigel	87	.P
181			4. Patterns that contain alternatives at the top level which do not all
182			start with the same pattern item may not work as expected. For example,
183			consider this pattern:
184			.sp
185			1234\|3789
186			.sp
187			If the first part of the subject is "ABC123", a partial match of the first
188			alternative is found at offset 3. There is no partial match for the second
189			alternative, because such a match does not start at the same point in the
190			subject string. Attempting to continue with the string "789" does not yield a
191			match because only those alternatives that match at one point in the subject
192			are remembered. The problem arises because the start of the second alternative
193			matches within the first alternative. There is no problem with anchored
194			patterns or patterns such as:
195			.sp
196			1234\|ABCD
197			.sp
198			where no string can be a partial match for both alternatives.
199	nigel	77	.
200			.
201	ph10	99	.SH AUTHOR
202			.rs
203			.sp
204			.nf
205			Philip Hazel
206			University Computing Service
207			Cambridge CB2 3QH, England.
208			.fi
209			.
210			.
211			.SH REVISION
212			.rs
213			.sp
214			.nf
215			Last updated: 06 March 2007
216			Copyright (c) 1997-2007 University of Cambridge.
217			.fi
Name	Value
svn:eol-style	native
svn:keywords	"Author Date Id Revision Url"