/[pcre]/code/trunk/maint/README
ViewVC logotype

Contents of /code/trunk/maint/README

Parent Directory Parent Directory | Revision Log Revision Log


Revision 181 - (hide annotations) (download)
Wed Jun 13 14:55:18 2007 UTC (7 years, 6 months ago) by ph10
File size: 11570 byte(s)
Documentation update preparatory to release.

1 ph10 122 MAINTENANCE README FOR PCRE
2     ---------------------------
3    
4 ph10 181 The files in the "maint" directory of the PCRE source contain data, scripts,
5 ph10 122 and programs that are used for the maintenance of PCRE, but which do not form
6 ph10 181 part of the PCRE distribution tarballs. This document describes these files and
7 ph10 122 also contains some notes for maintainers. Its contents are:
8 ph10 97
9 ph10 122 Files in the maint directory
10     Updating to a new Unicode release
11     Preparing for a PCRE release
12     Making a PCRE release
13 ph10 181 Long-term ideas (wish list)
14 ph10 122
15    
16     Files in the maint directory
17     ----------------------------
18    
19 ph10 129 Builducptable A Perl script that creates the contents of the ucptable.h file
20     from two Unicode data files, which themselves are downloaded
21     from the Unicode web site. Run this script in the "maint"
22     directory.
23 ph10 181
24 ph10 129 ManyConfigTests A shell script that runs "configure, make, test" a number of
25     times with different configuration settings.
26 ph10 181
27     Unicode.tables The files in this directory, Scripts.txt and UnicodeData.txt,
28     were downloaded from the Unicode web site. They contain
29 ph10 129 information about Unicode characters and scripts.
30 ph10 181
31 ph10 129 ucptest.c A short C program for testing the Unicode property functions
32     in pcre_ucp_searchfuncs.c, mainly useful after rebuilding the
33 ph10 181 Unicode property table. Compile and run this in the "maint"
34 ph10 129 directory.
35 ph10 181
36 ph10 129 ucptestdata A directory containing two files, testinput1 and testoutput1,
37     to use in conjunction with the ucptest program.
38 ph10 181
39 ph10 129 utf8.c A short, freestanding C program for converting a Unicode code
40     point into a sequence of bytes in the UTF-8 encoding, and vice
41     versa. If its argument is a hex number such as 0x1234, it
42     outputs a list of the equivalent UTF-8 bytes. If its argument
43     is sequence of concatenated UTF-8 bytes (e.g. e188b4) it
44     treats them as a UTF-8 character and outputs the equivalent
45     code point in hex.
46 ph10 97
47 ph10 181
48 ph10 122 Updating to a new Unicode release
49     ---------------------------------
50    
51 ph10 181 When there is a new release of Unicode, the files in Unicode.tables must be
52     refreshed from the web site, and the Buildupctable script can then be run to
53     generate a new version of ucptable.h. The ucptest program can be used to check
54     that the resulting table works properly, using the data files in ucptestdata to
55     check a number of test characters.
56 ph10 122
57    
58     Preparing for a PCRE release
59     ----------------------------
60    
61     This section contains a checklist of things that I consult before building a
62     distribution for a new release.
63    
64 ph10 155 . Ensure that the version number and version date are correct in configure.ac,
65     ChangeLog, and NEWS.
66 ph10 122
67     . Run ./autogen.sh to ensure everything is up-to-date.
68    
69 ph10 181 . Compile and test with many different config options, and combinations of
70 ph10 129 options. The maint/ManyConfigTests script now encapsulates this testing.
71 ph10 181
72 ph10 122 . Run perltest.pl on the test data for tests 1 and 4. The output should match
73 ph10 181 the PCRE test output, apart from the version identification at the top. The
74 ph10 122 other tests are not Perl-compatible (they use various special PCRE options).
75    
76     . Test with valgrind by running "RunTest valgrind". There is also "RunGrepTest
77     valgrind", though that takes quite a long time.
78 ph10 181
79     . It may also useful to test with Electric Fence, though the fact that it
80     grumbles for missing free() calls can be a nuisance. (A missing free() in
81 ph10 122 pcretest is hardly a big problem.) To build with EF, use:
82 ph10 181
83 ph10 122 LIBS='/usr/lib/libefence.a -lpthread' with ./configure.
84    
85     Then all normal runs use it to check for buffer overflow. Also run everything
86     with:
87 ph10 181
88     EF_PROTECT_BELOW=1 <whatever>
89    
90     because there have been problems with lookbehinds that looked too far.
91    
92     . Test with the emulated memmove() function by undefining HAVE_MEMMOVE and
93     HAVE_BCOPY in config.h. You may see a number of "pcre_memmove defined but not
94 ph10 155 used" warnings for the modules in which there is no call to memmove(). These
95     can be ignored.
96 ph10 122
97 ph10 181 . Documentation: check AUTHORS, COPYING, ChangeLog (check date), INSTALL,
98     LICENCE, NEWS (check date), NON-UNIX-USE, and README. Many of these won't
99 ph10 122 need changing, but over the long term things do change.
100 ph10 181
101 ph10 122 . Man pages: Check all man pages for \ not followed by e or f or " because
102     that indicates a markup error.
103    
104 ph10 181 . When the release is built, test it on a number of different operating
105 ph10 155 systems if possible, and using different compilers as well. For example,
106     on Solaris it is helpful to test using Sun's cc compiler as a change from
107     gcc. Adding -xarch=v9 to the cc options does a 64-bit test, but it also
108     needs -S 64 for pcretest to increase the stack size for test 2.
109 ph10 122
110 ph10 155
111 ph10 122 Making a PCRE release
112     ---------------------
113    
114     Run PrepareRelease and commit the files that it changes (by removing trailing
115 ph10 155 spaces). Then run "make distcheck" to create the tarballs and the zipball.
116 ph10 122
117     Don't forget to update Freshmeat when the new release is out, and to tell
118     webmaster@pcre.org and the mailing list.
119    
120    
121     Future ideas (wish list)
122     ------------------------
123    
124     This section records a list of ideas so that they do not get forgotten. They
125 ph10 181 vary enormously in their usefulness and potential for implementation. Some are
126 ph10 122 very sensible; some are rather wacky. Some have been on this list for years;
127     others are relatively new.
128    
129     . Optimization
130    
131 ph10 181 There are always ideas for new optimizations so as to speed up pattern
132     matching. Most of them try to save work by recognizing a non-match without
133 ph10 122 having to scan all the possibilities. These are some that I've recorded:
134    
135     * /((A{0,5}){0,5}){0,5}(something complex)/ on a non-matching string is very
136     slow, though Perl is fast. Can we speed up somehow? Convert to {0,125}?
137 ph10 181 OTOH, this is pathological - the user could easily fix it.
138    
139 ph10 122 * Turn ={4} into ==== ? (for speed). I once did an experiment, and it seems
140     to have little effect, and maybe makes things worse.
141 ph10 181
142     * "Ends with literal string" - note that a single character doesn't gain much
143 ph10 122 over the existing "required byte" (reqbyte) feature that just saves one
144     byte.
145 ph10 181
146 ph10 122 * These probably need to go in study():
147 ph10 181
148 ph10 122 o Remember an initial string rather than just 1 char?
149 ph10 181
150 ph10 122 o A required byte from alternatives - not just the last char, but an
151     earlier one if common to all alternatives.
152 ph10 181
153 ph10 122 o Minimum length of subject needed.
154 ph10 181
155 ph10 122 o Friedl contains other ideas.
156 ph10 181
157 ph10 122 . If Perl gets to a consistent state over the settings of capturing sub-
158     patterns inside repeats, see if we can match it. One example of the
159     difference is the matching of /(main(O)?)+/ against mainOmain, where PCRE
160     leaves $2 set. In Perl, it's unset. Changing this in PCRE will be very hard
161     because I think it needs much more state to be remembered.
162    
163 ph10 181 . Perl 6 will be a revolution. Is it a revolution too far for PCRE?
164    
165 ph10 122 . Unicode
166    
167 ph10 181 * Note that in Perl, \s matches \pZ and similarly for \d, \w and the POSIX
168     character classes. For the moment, I've chosen not to support this for
169     backward compatibility, for speed, and because it would be messy to
170 ph10 122 implement.
171 ph10 181
172 ph10 122 * A different approach to Unicode might be to use a typedef to do everything
173     in unsigned shorts instead of unsigned chars. Actually, we'd have to have a
174     new typedef to distinguish data from bits of compiled pattern that are in
175     bytes, I think. There would need to be conversion functions in and out. I
176     don't think this is particularly trivial - and anyway, Unicode now has
177     characters that need more than 16 bits, so is this at all sensible?
178 ph10 181
179 ph10 122 * There has been a request for direct support of 16-bit characters and
180     UTF-16. However, since Unicode is moving beyond purely 16-bit characters,
181     is this worth it at all? One possible way of handling 16-bit characters
182     would be to "load" them in the same way that UTF-8 characters are loaded.
183 ph10 181
184 ph10 122 . Allow errorptr and erroroffset to be NULL. I don't like this idea.
185    
186     . Line endings:
187    
188     * Option to use NUL as a line terminator in subject strings. This could now
189     be done relatively easily since the extension to support LF, CR, and CRLF.
190 ph10 181 If this is done, a suitable option for pcregrep is also required.
191    
192 ph10 122 . Option to provide the pattern with a length instead of with a NUL terminator.
193 ph10 181 This probably affects quite a few places in the code.
194 ph10 122
195 ph10 181 . Catch SIGSEGV for stack overflows?
196 ph10 122
197     . A feature to suspend a match via a callout was once requested.
198    
199     . Option to convert results into character offsets and character lengths.
200    
201 ph10 181 . Option for pcregrep to scan only the start of a file. I am not keen - this is
202 ph10 122 the job of "head".
203 ph10 181
204     . A (non-Unix) user wanted pcregrep options to (a) list a file name just once,
205 ph10 122 preceded by a blank line, instead of adding it to every matched line, and (b)
206     support --outputfile=name.
207 ph10 181
208 ph10 122 . Consider making UTF-8 and UCP the default for PCRE n.0 for some n > 7.
209    
210 ph10 181 . Add a user pointer to pcre_malloc/free functions -- some option would be
211 ph10 122 needed to retain backward compatibility.
212 ph10 181
213 ph10 122 . Define a union for the results from pcre_fullinfo().
214    
215 ph10 181 . Provide a "random access to the subject" facility so that the way in which it
216     is stored is independent of PCRE. For efficiency, it probably isn't possible
217 ph10 122 to switch this dynamically. It would have to be specified when PCRE was
218     compiled. PCRE would then call a function every time it wanted a character.
219 ph10 181
220 ph10 122 . There are new (*PRUNE) facilities in Perl 5.10, some of which it might be
221     relatively easy to implement.
222 ph10 181
223 ph10 122 . Wild thought: the ability to compile from PCRE's internal byte code to a real
224     FSM and a very fast (third) matcher to process the result. There would be
225     even more restrictions than for pcre_dfa_exec(), however. This is not easy.
226 ph10 181
227 ph10 122 . Should pcretest have some private locale data, to avoid relying on the
228     available locales for the test data, since different OS have different ideas?
229     This won't be as thorough a test, but perhaps that doesn't really matter.
230 ph10 181
231     . pcregrep: add -rs for a sorted recurse? Having to store file names and sort
232 ph10 122 them will of course slow it down.
233    
234 ph10 181 . Someone suggested --disable-callout to save code space when callouts are
235     never wanted. This seems rather marginal.
236 ph10 122
237 ph10 181 . "Cut" as described in Jeffrey Friedl's book, p364: \v and \V. The definitions
238     aren't yet clear enough for me. \v flushes saved states so that no
239     backtracking to anything earlier can happen; \V says "no more bumpalong", but
240     does it fail the current match? As described in the book, these aren't really
241     "cut" as in Prolog, are they? NOTE: (a) PCRE once had "cut", but it was
242     removed when atomic groups were introduced. (b) Perl 5.10 has some (*PRUNE)
243     features --
244 ph10 122
245 ph10 181 . These are the Perl 5.10 backtracking control features (all of which are
246     described as "experimental" -- some of them "very experimental") that it
247     might be easy to add to PCRE. They all succeed when encountered, but act as
248     follows when backtracking:
249    
250     (*PRUNE) fail this match attempt, but still bumpalong
251     (*SKIP) fail this match attempt, bumpalong to current match point
252     (*THEN) fail this branch, try next branch at same level or fail if none
253     (*COMMIT) fail this match attempt, suppress bumpalong
254     (*FAIL) fail and backtrack (same as (?!) and that can be optimized)
255     (*F) synonym for (*FAIL)
256     (*ACCEPT) behave as if end of pattern reached ("very experimental")
257    
258     Some of these can have arguments (*PRUNE:NAME) but I'm not sure whether they
259     make sense in the PCRE context.
260    
261 ph10 122 Philip Hazel
262     Email local part: ph10
263     Email domain: cam.ac.uk
264 ph10 181 Last updated: 13 June 2007

webmaster@exim.org
ViewVC Help
Powered by ViewVC 1.1.12