/[pcre]/code/trunk/ChangeLog
ViewVC logotype

Diff of /code/trunk/ChangeLog

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 63 by nigel, Sat Feb 24 21:40:03 2007 UTC revision 75 by nigel, Sat Feb 24 21:40:37 2007 UTC
# Line 1  Line 1 
1  ChangeLog for PCRE  ChangeLog for PCRE
2  ------------------  ------------------
3    
4  Version 4.00 17-Feb-03  Version 5.0 13-Sep-04
5  ----------------------  ---------------------
6    
7     1. Internal change: literal characters are no longer packed up into items
8        containing multiple characters in a single byte-string. Each character
9        is now matched using a separate opcode. However, there may be more than one
10        byte in the character in UTF-8 mode.
11    
12     2. The pcre_callout_block structure has two new fields: pattern_position and
13        next_item_length. These contain the offset in the pattern to the next match
14        item, and its length, respectively.
15    
16     3. The PCRE_AUTO_CALLOUT option for pcre_compile() requests the automatic
17        insertion of callouts before each pattern item. Added the /C option to
18        pcretest to make use of this.
19    
20     4. On the advice of a Windows user, the lines
21    
22          #if defined(_WIN32) || defined(WIN32)
23          _setmode( _fileno( stdout ), 0x8000 );
24          #endif  /* defined(_WIN32) || defined(WIN32) */
25    
26        have been added to the source of pcretest. This apparently does useful
27        magic in relation to line terminators.
28    
29     5. Changed "r" and "w" in the calls to fopen() in pcretest to "rb" and "wb"
30        for the benefit of those environments where the "b" makes a difference.
31    
32     6. The icc compiler has the same options as gcc, but "configure" doesn't seem
33        to know about it. I have put a hack into configure.in that adds in code
34        to set GCC=yes if CC=icc. This seems to end up at a point in the
35        generated configure script that is early enough to affect the setting of
36        compiler options, which is what is needed, but I have no means of testing
37        whether it really works. (The user who reported this had patched the
38        generated configure script, which of course I cannot do.)
39    
40        LATER: After change 22 below (new libtool files), the configure script
41        seems to know about icc (and also ecc). Therefore, I have commented out
42        this hack in configure.in.
43    
44     7. Added support for pkg-config (2 patches were sent in).
45    
46     8. Negated POSIX character classes that used a combination of internal tables
47        were completely broken. These were [[:^alpha:]], [[:^alnum:]], and
48        [[:^ascii]]. Typically, they would match almost any characters. The other
49        POSIX classes were not broken in this way.
50    
51     9. Matching the pattern "\b.*?" against "ab cd", starting at offset 1, failed
52        to find the match, as PCRE was deluded into thinking that the match had to
53        start at the start point or following a newline. The same bug applied to
54        patterns with negative forward assertions or any backward assertions
55        preceding ".*" at the start, unless the pattern required a fixed first
56        character. This was a failing pattern: "(?!.bcd).*". The bug is now fixed.
57    
58    10. In UTF-8 mode, when moving forwards in the subject after a failed match
59        starting at the last subject character, bytes beyond the end of the subject
60        string were read.
61    
62    11. Renamed the variable "class" as "classbits" to make life easier for C++
63        users. (Previously there was a macro definition, but it apparently wasn't
64        enough.)
65    
66    12. Added the new field "tables" to the extra data so that tables can be passed
67        in at exec time, or the internal tables can be re-selected. This allows
68        a compiled regex to be saved and re-used at a later time by a different
69        program that might have everything at different addresses.
70    
71    13. Modified the pcre-config script so that, when run on Solaris, it shows a
72        -R library as well as a -L library.
73    
74    14. The debugging options of pcretest (-d on the command line or D on a
75        pattern) showed incorrect output for anything following an extended class
76        that contained multibyte characters and which was followed by a quantifier.
77    
78    15. Added optional support for general category Unicode character properties
79        via the \p, \P, and \X escapes. Unicode property support implies UTF-8
80        support. It adds about 90K to the size of the library. The meanings of the
81        inbuilt class escapes such as \d and \s have NOT been changed.
82    
83    16. Updated pcredemo.c to include calls to free() to release the memory for the
84        compiled pattern.
85    
86    17. The generated file chartables.c was being created in the source directory
87        instead of in the building directory. This caused the build to fail if the
88        source directory was different from the building directory, and was
89        read-only.
90    
91    18. Added some sample Win commands from Mark Tetrode into the NON-UNIX-USE
92        file. No doubt somebody will tell me if they don't make sense... Also added
93        Dan Mooney's comments about building on OpenVMS.
94    
95    19. Added support for partial matching via the PCRE_PARTIAL option for
96        pcre_exec() and the \P data escape in pcretest.
97    
98    20. Extended pcretest with 3 new pattern features:
99    
100        (i)   A pattern option of the form ">rest-of-line" causes pcretest to
101              write the compiled pattern to the file whose name is "rest-of-line".
102              This is a straight binary dump of the data, with the saved pointer to
103              the character tables forced to be NULL. The study data, if any, is
104              written too. After writing, pcretest reads a new pattern.
105    
106        (ii)  If, instead of a pattern, "<rest-of-line" is given, pcretest reads a
107              compiled pattern from the given file. There must not be any
108              occurrences of "<" in the file name (pretty unlikely); if there are,
109              pcretest will instead treat the initial "<" as a pattern delimiter.
110              After reading in the pattern, pcretest goes on to read data lines as
111              usual.
112    
113        (iii) The F pattern option causes pcretest to flip the bytes in the 32-bit
114              and 16-bit fields in a compiled pattern, to simulate a pattern that
115              was compiled on a host of opposite endianness.
116    
117    21. The pcre-exec() function can now cope with patterns that were compiled on
118        hosts of opposite endianness, with this restriction:
119    
120          As for any compiled expression that is saved and used later, the tables
121          pointer field cannot be preserved; the extra_data field in the arguments
122          to pcre_exec() should be used to pass in a tables address if a value
123          other than the default internal tables were used at compile time.
124    
125    22. Calling pcre_exec() with a negative value of the "ovecsize" parameter is
126        now diagnosed as an error. Previously, most of the time, a negative number
127        would have been treated as zero, but if in addition "ovector" was passed as
128        NULL, a crash could occur.
129    
130    23. Updated the files ltmain.sh, config.sub, config.guess, and aclocal.m4 with
131        new versions from the libtool 1.5 distribution (the last one is a copy of
132        a file called libtool.m4). This seems to have fixed the need to patch
133        "configure" to support Darwin 1.3 (which I used to do). However, I still
134        had to patch ltmain.sh to ensure that ${SED} is set (it isn't on my
135        workstation).
136    
137    24. Changed the PCRE licence to be the more standard "BSD" licence.
138    
139    
140    Version 4.5 01-Dec-03
141    ---------------------
142    
143     1. There has been some re-arrangement of the code for the match() function so
144        that it can be compiled in a version that does not call itself recursively.
145        Instead, it keeps those local variables that need separate instances for
146        each "recursion" in a frame on the heap, and gets/frees frames whenever it
147        needs to "recurse". Keeping track of where control must go is done by means
148        of setjmp/longjmp. The whole thing is implemented by a set of macros that
149        hide most of the details from the main code, and operates only if
150        NO_RECURSE is defined while compiling pcre.c. If PCRE is built using the
151        "configure" mechanism, "--disable-stack-for-recursion" turns on this way of
152        operating.
153    
154        To make it easier for callers to provide specially tailored get/free
155        functions for this usage, two new functions, pcre_stack_malloc, and
156        pcre_stack_free, are used. They are always called in strict stacking order,
157        and the size of block requested is always the same.
158    
159        The PCRE_CONFIG_STACKRECURSE info parameter can be used to find out whether
160        PCRE has been compiled to use the stack or the heap for recursion. The
161        -C option of pcretest uses this to show which version is compiled.
162    
163        A new data escape \S, is added to pcretest; it causes the amounts of store
164        obtained and freed by both kinds of malloc/free at match time to be added
165        to the output.
166    
167     2. Changed the locale test to use "fr_FR" instead of "fr" because that's
168        what's available on my current Linux desktop machine.
169    
170     3. When matching a UTF-8 string, the test for a valid string at the start has
171        been extended. If start_offset is not zero, PCRE now checks that it points
172        to a byte that is the start of a UTF-8 character. If not, it returns
173        PCRE_ERROR_BADUTF8_OFFSET (-11). Note: the whole string is still checked;
174        this is necessary because there may be backward assertions in the pattern.
175        When matching the same subject several times, it may save resources to use
176        PCRE_NO_UTF8_CHECK on all but the first call if the string is long.
177    
178     4. The code for checking the validity of UTF-8 strings has been tightened so
179        that it rejects (a) strings containing 0xfe or 0xff bytes and (b) strings
180        containing "overlong sequences".
181    
182     5. Fixed a bug (appearing twice) that I could not find any way of exploiting!
183        I had written "if ((digitab[*p++] && chtab_digit) == 0)" where the "&&"
184        should have been "&", but it just so happened that all the cases this let
185        through by mistake were picked up later in the function.
186    
187     6. I had used a variable called "isblank" - this is a C99 function, causing
188        some compilers to warn. To avoid this, I renamed it (as "blankclass").
189    
190     7. Cosmetic: (a) only output another newline at the end of pcretest if it is
191        prompting; (b) run "./pcretest /dev/null" at the start of the test script
192        so the version is shown; (c) stop "make test" echoing "./RunTest".
193    
194     8. Added patches from David Burgess to enable PCRE to run on EBCDIC systems.
195    
196     9. The prototype for memmove() for systems that don't have it was using
197        size_t, but the inclusion of the header that defines size_t was later. I've
198        moved the #includes for the C headers earlier to avoid this.
199    
200    10. Added some adjustments to the code to make it easier to compiler on certain
201        special systems:
202    
203          (a) Some "const" qualifiers were missing.
204          (b) Added the macro EXPORT before all exported functions; by default this
205              is defined to be empty.
206          (c) Changed the dftables auxiliary program (that builds chartables.c) so
207              that it reads its output file name as an argument instead of writing
208              to the standard output and assuming this can be redirected.
209    
210    11. In UTF-8 mode, if a recursive reference (e.g. (?1)) followed a character
211        class containing characters with values greater than 255, PCRE compilation
212        went into a loop.
213    
214    12. A recursive reference to a subpattern that was within another subpattern
215        that had a minimum quantifier of zero caused PCRE to crash. For example,
216        (x(y(?2))z)? provoked this bug with a subject that got as far as the
217        recursion. If the recursively-called subpattern itself had a zero repeat,
218        that was OK.
219    
220    13. In pcretest, the buffer for reading a data line was set at 30K, but the
221        buffer into which it was copied (for escape processing) was still set at
222        1024, so long lines caused crashes.
223    
224    14. A pattern such as /[ab]{1,3}+/ failed to compile, giving the error
225        "internal error: code overflow...". This applied to any character class
226        that was followed by a possessive quantifier.
227    
228    15. Modified the Makefile to add libpcre.la as a prerequisite for
229        libpcreposix.la because I was told this is needed for a parallel build to
230        work.
231    
232    16. If a pattern that contained .* following optional items at the start was
233        studied, the wrong optimizing data was generated, leading to matching
234        errors. For example, studying /[ab]*.*c/ concluded, erroneously, that any
235        matching string must start with a or b or c. The correct conclusion for
236        this pattern is that a match can start with any character.
237    
238    
239    Version 4.4 13-Aug-03
240    ---------------------
241    
242     1. In UTF-8 mode, a character class containing characters with values between
243        127 and 255 was not handled correctly if the compiled pattern was studied.
244        In fixing this, I have also improved the studying algorithm for such
245        classes (slightly).
246    
247     2. Three internal functions had redundant arguments passed to them. Removal
248        might give a very teeny performance improvement.
249    
250     3. Documentation bug: the value of the capture_top field in a callout is *one
251        more than* the number of the hightest numbered captured substring.
252    
253     4. The Makefile linked pcretest and pcregrep with -lpcre, which could result
254        in incorrectly linking with a previously installed version. They now link
255        explicitly with libpcre.la.
256    
257     5. configure.in no longer needs to recognize Cygwin specially.
258    
259     6. A problem in pcre.in for Windows platforms is fixed.
260    
261     7. If a pattern was successfully studied, and the -d (or /D) flag was given to
262        pcretest, it used to include the size of the study block as part of its
263        output. Unfortunately, the structure contains a field that has a different
264        size on different hardware architectures. This meant that the tests that
265        showed this size failed. As the block is currently always of a fixed size,
266        this information isn't actually particularly useful in pcretest output, so
267        I have just removed it.
268    
269     8. Three pre-processor statements accidentally did not start in column 1.
270        Sadly, there are *still* compilers around that complain, even though
271        standard C has not required this for well over a decade. Sigh.
272    
273     9. In pcretest, the code for checking callouts passed small integers in the
274        callout_data field, which is a void * field. However, some picky compilers
275        complained about the casts involved for this on 64-bit systems. Now
276        pcretest passes the address of the small integer instead, which should get
277        rid of the warnings.
278    
279    10. By default, when in UTF-8 mode, PCRE now checks for valid UTF-8 strings at
280        both compile and run time, and gives an error if an invalid UTF-8 sequence
281        is found. There is a option for disabling this check in cases where the
282        string is known to be correct and/or the maximum performance is wanted.
283    
284    11. In response to a bug report, I changed one line in Makefile.in from
285    
286            -Wl,--out-implib,.libs/lib@WIN_PREFIX@pcreposix.dll.a \
287        to
288            -Wl,--out-implib,.libs/@WIN_PREFIX@libpcreposix.dll.a \
289    
290        to look similar to other lines, but I have no way of telling whether this
291        is the right thing to do, as I do not use Windows. No doubt I'll get told
292        if it's wrong...
293    
294    
295    Version 4.3 21-May-03
296    ---------------------
297    
298    1. Two instances of @WIN_PREFIX@ omitted from the Windows targets in the
299       Makefile.
300    
301    2. Some refactoring to improve the quality of the code:
302    
303       (i)   The utf8_table... variables are now declared "const".
304    
305       (ii)  The code for \cx, which used the "case flipping" table to upper case
306             lower case letters, now just substracts 32. This is ASCII-specific,
307             but the whole concept of \cx is ASCII-specific, so it seems
308             reasonable.
309    
310       (iii) PCRE was using its character types table to recognize decimal and
311             hexadecimal digits in the pattern. This is silly, because it handles
312             only 0-9, a-f, and A-F, but the character types table is locale-
313             specific, which means strange things might happen. A private
314             table is now used for this - though it costs 256 bytes, a table is
315             much faster than multiple explicit tests. Of course, the standard
316             character types table is still used for matching digits in subject
317             strings against \d.
318    
319       (iv)  Strictly, the identifier ESC_t is reserved by POSIX (all identifiers
320             ending in _t are). So I've renamed it as ESC_tee.
321    
322    3. The first argument for regexec() in the POSIX wrapper should have been
323       defined as "const".
324    
325    4. Changed pcretest to use malloc() for its buffers so that they can be
326       Electric Fenced for debugging.
327    
328    5. There were several places in the code where, in UTF-8 mode, PCRE would try
329       to read one or more bytes before the start of the subject string. Often this
330       had no effect on PCRE's behaviour, but in some circumstances it could
331       provoke a segmentation fault.
332    
333    6. A lookbehind at the start of a pattern in UTF-8 mode could also cause PCRE
334       to try to read one or more bytes before the start of the subject string.
335    
336    7. A lookbehind in a pattern matched in non-UTF-8 mode on a PCRE compiled with
337       UTF-8 support could misbehave in various ways if the subject string
338       contained bytes with the 0x80 bit set and the 0x40 bit unset in a lookbehind
339       area. (PCRE was not checking for the UTF-8 mode flag, and trying to move
340       back over UTF-8 characters.)
341    
342    
343    Version 4.2 14-Apr-03
344    ---------------------
345    
346    1. Typo "#if SUPPORT_UTF8" instead of "#ifdef SUPPORT_UTF8" fixed.
347    
348    2. Changes to the building process, supplied by Ronald Landheer-Cieslak
349         [ON_WINDOWS]: new variable, "#" on non-Windows platforms
350         [NOT_ON_WINDOWS]: new variable, "#" on Windows platforms
351         [WIN_PREFIX]: new variable, "cyg" for Cygwin
352         * Makefile.in: use autoconf substitution for OBJEXT, EXEEXT, BUILD_OBJEXT
353           and BUILD_EXEEXT
354         Note: automatic setting of the BUILD variables is not yet working
355         set CPPFLAGS and BUILD_CPPFLAGS (but don't use yet) - should be used at
356           compile-time but not at link-time
357         [LINK]: use for linking executables only
358         make different versions for Windows and non-Windows
359         [LINKLIB]: new variable, copy of UNIX-style LINK, used for linking
360           libraries
361         [LINK_FOR_BUILD]: new variable
362         [OBJEXT]: use throughout
363         [EXEEXT]: use throughout
364         <winshared>: new target
365         <wininstall>: new target
366         <dftables.o>: use native compiler
367         <dftables>: use native linker
368         <install>: handle Windows platform correctly
369         <clean>: ditto
370         <check>: ditto
371         copy DLL to top builddir before testing
372    
373       As part of these changes, -no-undefined was removed again. This was reported
374       to give trouble on HP-UX 11.0, so getting rid of it seems like a good idea
375       in any case.
376    
377    3. Some tidies to get rid of compiler warnings:
378    
379       . In the match_data structure, match_limit was an unsigned long int, whereas
380         match_call_count was an int. I've made them both unsigned long ints.
381    
382       . In pcretest the fact that a const uschar * doesn't automatically cast to
383         a void * provoked a warning.
384    
385       . Turning on some more compiler warnings threw up some "shadow" variables
386         and a few more missing casts.
387    
388    4. If PCRE was complied with UTF-8 support, but called without the PCRE_UTF8
389       option, a class that contained a single character with a value between 128
390       and 255 (e.g. /[\xFF]/) caused PCRE to crash.
391    
392    5. If PCRE was compiled with UTF-8 support, but called without the PCRE_UTF8
393       option, a class that contained several characters, but with at least one
394       whose value was between 128 and 255 caused PCRE to crash.
395    
396    
397    Version 4.1 12-Mar-03
398    ---------------------
399    
400    1. Compiling with gcc -pedantic found a couple of places where casts were
401    needed, and a string in dftables.c that was longer than standard compilers are
402    required to support.
403    
404    2. Compiling with Sun's compiler found a few more places where the code could
405    be tidied up in order to avoid warnings.
406    
407    3. The variables for cross-compiling were called HOST_CC and HOST_CFLAGS; the
408    first of these names is deprecated in the latest Autoconf in favour of the name
409    CC_FOR_BUILD, because "host" is typically used to mean the system on which the
410    compiled code will be run. I can't find a reference for HOST_CFLAGS, but by
411    analogy I have changed it to CFLAGS_FOR_BUILD.
412    
413    4. Added -no-undefined to the linking command in the Makefile, because this is
414    apparently helpful for Windows. To make it work, also added "-L. -lpcre" to the
415    linking step for the pcreposix library.
416    
417    5. PCRE was failing to diagnose the case of two named groups with the same
418    name.
419    
420    6. A problem with one of PCRE's optimizations was discovered. PCRE remembers a
421    literal character that is needed in the subject for a match, and scans along to
422    ensure that it is present before embarking on the full matching process. This
423    saves time in cases of nested unlimited repeats that are never going to match.
424    Problem: the scan can take a lot of time if the subject is very long (e.g.
425    megabytes), thus penalizing straightforward matches. It is now done only if the
426    amount of subject to be scanned is less than 1000 bytes.
427    
428    7. A lesser problem with the same optimization is that it was recording the
429    first character of an anchored pattern as "needed", thus provoking a search
430    right along the subject, even when the first match of the pattern was going to
431    fail. The "needed" character is now not set for anchored patterns, unless it
432    follows something in the pattern that is of non-fixed length. Thus, it still
433    fulfils its original purpose of finding quick non-matches in cases of nested
434    unlimited repeats, but isn't used for simple anchored patterns such as /^abc/.
435    
436    
437    Version 4.0 17-Feb-03
438    ---------------------
439    
440  1. If a comment in an extended regex that started immediately after a meta-item  1. If a comment in an extended regex that started immediately after a meta-item
441  extended to the end of string, PCRE compiled incorrect data. This could lead to  extended to the end of string, PCRE compiled incorrect data. This could lead to

Legend:
Removed from v.63  
changed lines
  Added in v.75

webmaster@exim.org
ViewVC Help
Powered by ViewVC 1.1.12