summaryrefslogtreecommitdiff
path: root/xdelta3
diff options
context:
space:
mode:
Diffstat (limited to 'xdelta3')
-rwxr-xr-xxdelta3/Makefile2
-rwxr-xr-xxdelta3/setup.py2
-rwxr-xr-xxdelta3/www/xdelta3-api-guide.html212
-rwxr-xr-xxdelta3/www/xdelta3-cmdline.html167
-rwxr-xr-xxdelta3/www/xdelta3.css69
-rwxr-xr-xxdelta3/www/xdelta3.html89
-rwxr-xr-xxdelta3/xdelta3-main.h2
-rwxr-xr-xxdelta3/xdelta3-regtest.py152
-rwxr-xr-xxdelta3/xdelta3.prj43
9 files changed, 79 insertions, 659 deletions
diff --git a/xdelta3/Makefile b/xdelta3/Makefile
index 574ff60..b71e24a 100755
--- a/xdelta3/Makefile
+++ b/xdelta3/Makefile
@@ -38,7 +38,7 @@ EXTRA = Makefile COPYING linkxd3lib.c badcopy.c www \
38 draft-korn-vcdiff.txt xdelta3.vcproj badcopy.vcproj 38 draft-korn-vcdiff.txt xdelta3.vcproj badcopy.vcproj
39 39
40# $Format: "REL=$Xdelta3Version$" $ 40# $Format: "REL=$Xdelta3Version$" $
41REL=0m 41REL=0n
42RELDIR = xdelta3$(REL) 42RELDIR = xdelta3$(REL)
43 43
44all: xdelta3-debug xdelta3 $(PYTGT) 44all: xdelta3-debug xdelta3 $(PYTGT)
diff --git a/xdelta3/setup.py b/xdelta3/setup.py
index ed1e1aa..faee42b 100755
--- a/xdelta3/setup.py
+++ b/xdelta3/setup.py
@@ -52,7 +52,7 @@ xdelta3_ext = Extension('xdelta3',
52 ]) 52 ])
53 53
54# $Format: "REL='$Xdelta3Version$'" $ 54# $Format: "REL='$Xdelta3Version$'" $
55REL='0m' 55REL='0n'
56 56
57# This provides xdelta3.main(), which calls the xdelta3 command-line main() 57# This provides xdelta3.main(), which calls the xdelta3 command-line main()
58# from python. 58# from python.
diff --git a/xdelta3/www/xdelta3-api-guide.html b/xdelta3/www/xdelta3-api-guide.html
deleted file mode 100755
index b3513ea..0000000
--- a/xdelta3/www/xdelta3-api-guide.html
+++ /dev/null
@@ -1,212 +0,0 @@
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2<html>
3<head>
4 <title>Xdelta3 API guide (BETA)</title>
5 <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
6 <link rel="stylesheet" type="text/css" href="xdelta3.css"/>
7</head>
8<body>
9
10<!-- $Format: "$WWWLeftNavBar$" $ --!>
11<table cellpadding="20px" width=700> <tr> <td class="leftbdr" valign=top height=600 width=100> <div class="leftbody"> <h1>Xdelta</h1> <a href="xdelta3.html">overview</a><br> <a href="xdelta3-cmdline.html">command&nbsp;line</a><br> <a href="xdelta3-api-guide.html">api&nbsp;guide</a><br> <br><a href="http://xdelta.org">xdelta.org</a></h2> </div> </td> <td valign=top width=500>
12
13
14<!-- Copyright (C) 2003 and onward. Joshua P. MacDonald --!>
15
16<h1>api guide</h1>
17
18<p>This guide intends to give you a quick start to the Xdelta3 programming
19interface. This is not a complete reference, the comments inside source file
20<code>xdelta3.h</code> and the command-line application,
21<code>xdelta3-main.h</code> offer more complete information.</p>
22
23<p>Have you read the <a href="xdelta3-cmdline.html">command-line interface</a>?</p>
24
25<h1>stream interface</h1>
26
27<p>
28To begin with, there are three external structures, only two of which are
29discussed here. The <code>xd3_stream</code> struct plays the main role, one
30of these contains the state necessary to encode or decode one stream of data.
31An <code>xd3_source</code> struct maintains state about the (optional) source
32file, against which differences are computed. The third structure,
33<code>xd3_config</code> deals with configuring various encoder parameters.</p>
34
35<p>
36At a glance, the interface resembles Zlib. The program puts data in, which
37the xd3_stream consumes. After computing over the data, the xd3_stream in
38turn generates output for the application to consume, or it requests more
39input. The xd3_stream also issues requests to the application to read a block
40of source data. The request to read a source block may be handled in one of
41two ways, according to application preference. If a <code>xd3_getblk</code>
42callback function is provided, the application handler will be called from
43within the library, suspending computation until the request completes. If no
44callback function is provided the library returns a special code
45(XD3_GETSRCBLK), allowing the application to issue the request and resume
46computation whenever it likes. In both cases, the xd3_source struct contains
47the requested block number and a place to store the result.</p>
48
49<h1>setup</h1>
50<p>The code to declare and initialize the xd3_stream:</p>
51<div class="example">
52<pre>
53int ret;
54xd3_stream stream;
55xd3_config config;
56
57xd3_init_config (&config, 0 /* flags */);
58config.winsize = 32768;
59ret = xd3_config_stream (&stream, &config);
60
61if (ret != 0) { /* error */ }
62</pre>
63</div>
64
65<p>
66<code>xd3_init_config()</code> initializes the <code>xd3_config</code> struct
67with default values. Many settings remain undocumented in the beta release.
68The most relevant setting, <code>xd3_config.winsize</code>, sets the encoder
69window size. The encoder allocates a buffer of this size if the program
70supplies input in smaller units (unless the <code>XD3_FLUSH</code> flag is
71set). <code>xd3_config_stream()</code> initializes the <code>xd3_stream</code>
72object with the supplied configuration.
73</p>
74
75<h1>setting the source</h1>
76<p>
77The stream is ready for input at this point, though for encoding the source
78data must be supplied now. To declare an initialize the xd3_source:</p>
79
80<div class="example">
81<pre>
82xd3_source source;
83void *IO_handle = ...;
84
85source.name = "...";
86source.size = file_size;
87source.ioh= IO_handle;
88source.blksize= 32768;
89source.curblkno = (xoff_t) -1;
90source.curblk = NULL;
91
92ret = xd3_set_source (&stream, &source);
93
94if (ret != 0) { /* error */ }
95</pre>
96</div>
97
98<p>
99The decoder sets source data in the same manner, but it may delay this step
100until the application header has been received (<code>XD3_GOTHEADER</code>).
101The application can also check whether source data is required for decoding
102with the <code>xd3_decoder_needs_source()</code>.</p>
103
104<p>
105<code>xd3_source.blksize</code> determines the block size used for requesting
106source blocks. If the first source block (or the entire source) is already in
107memory, set <code>curblkno</code> to 0 and <code>curblk</code> to that block
108of data.</p>
109
110<h1>input/output loop</h1>
111
112<p>The stream is now ready for input, which the application provides by
113calling <code>xd3_avail_input()</code>. The application initiates
114encoding or decoding at this point by calling one of two functions:</p>
115
116<div class="example">
117<pre>
118int xd3_encode_input (xd3_stream *stream)
119int xd3_decode_input (xd3_stream *stream)
120</pre>
121</div>
122
123<p>Unless there is an error, these routines return one of six result
124codes which the application must handle. In many cases, all or most
125of the handler code is shared between encoding and decoding. The
126codes are:</p>
127
128<ul>
129<li> <code>XD3_INPUT</code>: The stream is ready for (or requires) more input. The
130application should call xd3_avail_input when (if) more data is
131available.
132
133<li> <code>XD3_OUTPUT</code>: The stream has pending output. The application
134should write or otherwise consume the block of data found in the
135xd3_stream fields <code>next_out</code> and <code>avail_out</code>,
136then call <code>xd3_consume_output</code>.
137
138<li> <code>XD3_GETSRCBLK</code>: The stream is requesting a source block be read,
139as described above. This is only ever returned if the xd3_getblk
140callback was not provided.
141
142<li> <code>XD3_GOTHEADER</code>: This decoder-specific code indicates that the
143first VCDIFF window header has been received. This gives the
144application a chance to inspect the application header before
145encoding the first window.
146
147<li> <code>XD3_WINSTART</code>: This is returned by both encoder and decoder prior to
148processing a window. For encoding, this code is returned once there is enough
149available input. For decoding, this is returned following each window header
150(except the first, when XD3_GOTHEADER is returned instead).
151
152<li> <code>XD3_WINFINISH</code>: This is called when the output from a single
153window has been fully consumed.
154</ul>
155
156<p>An application could be structured something like this:</p>
157
158<div class="example">
159<pre>
160do {
161 read (&indata, &insize);
162 if (reached_EOF) {
163 xd3_set_flags (&stream, XD3_FLUSH);
164 }
165 xd3_avail_input (&stream, indata, insize);
166process:
167 ret = xd3_xxcode_input (&stream);
168 switch (ret) {
169 case XD3_INPUT:
170 continue;
171 case XD3_OUTPUT:
172 /* write data */
173 goto process;
174 case XD3_GETSRCBLK:
175 /* set source block */
176 goto process;
177 case XD3_GOTHEADER:
178 case XD3_WINSTART:
179 case XD3_WINFINISH:
180 /* no action necessary */
181 goto process;
182 default:
183 /* error */
184 }
185} while (! reached_EOF);
186</pre>
187</div>
188
189<p>
190All that remains is to close the stream and free its resources. The
191<code>xd3_close_stream()</code> checks several error conditions but otherwise
192involves no input or output. The <code>xd3_free_stream()</code> routine frees
193all memory allocated by the stream.</p>
194
195<h1>misc</h1>
196
197<p>
198There are two convenience functions for encoding to and decoding from
199in-memory buffers. See the <code>xd3_encode_completely</code> and
200<code>xd3_decode_completely</code> interfaces.</p>
201
202<p>
203There are two routines to get and set the application header. When
204encoding, sthe application header must be set before the first
205<code>XD3_WINSTART</code>. When decoding, the application header is available
206after after the first <code>XD3_GOTHEADER</code>.</p>
207
208</td>
209</tr>
210</table>
211</body>
212</html>
diff --git a/xdelta3/www/xdelta3-cmdline.html b/xdelta3/www/xdelta3-cmdline.html
deleted file mode 100755
index e5203a4..0000000
--- a/xdelta3/www/xdelta3-cmdline.html
+++ /dev/null
@@ -1,167 +0,0 @@
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2<html>
3<head>
4 <title>Xdelta3 command-line guide (BETA)</title>
5 <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
6 <link rel="stylesheet" type="text/css" href="xdelta3.css"/>
7</head>
8<body>
9
10<!-- $Format: "$WWWLeftNavBar$" $ --!>
11<table cellpadding="20px" width=700> <tr> <td class="leftbdr" valign=top height=600 width=100> <div class="leftbody"> <h1>Xdelta</h1> <a href="xdelta3.html">overview</a><br> <a href="xdelta3-cmdline.html">command&nbsp;line</a><br> <a href="xdelta3-api-guide.html">api&nbsp;guide</a><br> <br><a href="http://xdelta.org">xdelta.org</a></h2> </div> </td> <td valign=top width=500>
12
13
14<!-- Copyright (C) 2003 and onward. Joshua P. MacDonald --!>
15
16<h1>command-line guide</h1>
17
18<code>xdelta3</code> can be run with syntax familiar but not similar to gzip;
19it requires you to specify the output file in most cases, rather than applying
20any default filename extensions. These are cases that resemble gzip:<p>
21
22<div class="example">
23<pre>
24xdelta3 -c file_to_compress > delta_file
25xdelta3 -dc delta_file > file_uncompressed
26</pre>
27</div>
28<p>
29
30The <code>-c</code> option says to write to the standard output. The
31<code>-d</code> option says to decode. The default action is to encode (also
32specified by <code>-e</code>). <code>xdelta3</code> also supports long
33command names, these two commands are equivalent to the ones abvove:<p>
34
35<div class="example">
36<pre>
37xdelta3 encode file_to_compress > delta_file
38xdelta3 decode delta_file > file_uncompressed
39</pre>
40</div>
41<p>
42
43<code>xdelta3</code> has the notion of a default filename for decoding. If
44you specified a file name during the encode step, it is used as the default
45for decoding. The <code>-s</code> option specifies a <em>source file</em> for
46delta-compression.<p>
47
48<div class="example">
49<pre>
50xdelta3 -s source_file target_file delta_file
51xdelta3 -d delta_file
52</pre>
53</div>
54<p>
55
56The second line above fills in "source_file" and "target_file" as the input
57and output filenames. Without the <code>-f</code> option,
58<code>xdelta3</code> will not overwrite an existing file. When there are no
59default filenames (e.g., in decode), standard input and standard output are
60used. In the example below, the default source filename is applied in
61decoding.
62<p>
63
64<div class="example">
65<pre>
66cat target_file | xdelta3 -s source_file > delta_file
67xdelta3 -d < delta_file > target_file.1
68</pre>
69</div>
70<p>
71
72<code>xdelta3</code> recognizes externally compressed inputs, so the following
73command produces the expected results:<p>
74
75<div class="example">
76<pre>
77xdelta3 -s beta2.tar.gz beta3.tar.gz beta3.tar.gz.xd
78xdelta3 -ds beta2.tar.gz beta3.tar.gz.xd beta3.tar.gz.1
79</pre>
80</div>
81<p>
82
83You can avoid the intermediate file and use <code>xdelta3</code> together
84with a <code>tar</code>-pipeline.
85
86<div class="example">
87<pre>
88tar -cf - beta3 | xdelta3 -s beta2.tar > beta3.tar.xd
89xdelta3 -d beta3.tar.xd | tar -xf -
90</pre>
91</div>
92<p>
93
94<code>xdelta</code> can print various information about a compressed file with
95the "printhdr" command. The "printhdrs" command prints information about each
96<em>window</em> of the encoding. The "printdelta" command prints the actual
97encoding for each window, in human-readable format.<p>
98
99<div class="example">
100<pre>
101# xdelta3 printdelta delta_file
102VCDIFF version: 0
103VCDIFF header size: 5
104VCDIFF header indicator: none
105VCDIFF secondary compressor: none
106VCDIFF window number: 0
107VCDIFF window indicator: VCD_SOURCE VCD_ADLER32
108VCDIFF adler32 checksum: 48BFADB6
109VCDIFF copy window length: 2813
110VCDIFF copy window offset: 0
111VCDIFF delta encoding length: 93
112VCDIFF target window length: 2903
113VCDIFF data section length: 72
114VCDIFF inst section length: 8
115VCDIFF addr section length: 3
116 Offset Code Type1 Size1 @Addr1 + Type2 Size2 @Addr2
117 000000 019 CPY_0 1535 @0
118 001535 001 ADD 72
119 001607 019 CPY_0 1296 @1517
120</pre>
121</div>
122<br>
123<p>
124
125
126<h1>xdelta3 -h</h1>
127
128<pre>
129usage: xdelta3 [command/options] [input [output]]
130special command names:
131 config prints xdelta3 configuration
132 decode decompress the input
133 encode compress the input
134 test run the builtin tests
135special commands for VCDIFF inputs:
136 printdelta print information about the entire delta
137 printhdr print information about the first window
138 printhdrs print information about all windows
139standard options:
140 -0 .. -9 compression level
141 -c use stdout
142 -d decompress
143 -e compress
144 -f force overwrite
145 -h show help
146 -q be quiet
147 -v be verbose (max 2)
148 -V show version
149memory options:
150 -B blksize source file block size
151 -M memsize memory budget for hash tables
152 -W winsize input window buffer size
153compression options:
154 -s source source file to copy from (if any)
155 -S [djw|fgk] enable/disable secondary compression
156 -N disable small string-matching compression
157 -D disable external decompression (encode/decode)
158 -R disable external recompression (decode)
159</pre>
160<p>
161
162</td>
163</tr>
164</table>
165
166</body>
167</html>
diff --git a/xdelta3/www/xdelta3.css b/xdelta3/www/xdelta3.css
deleted file mode 100755
index 269b1c9..0000000
--- a/xdelta3/www/xdelta3.css
+++ /dev/null
@@ -1,69 +0,0 @@
1body {
2 margin-top: 15px;
3 margin-left: 15px;
4 background-color:#b0b0b0;
5 color:#204080;
6 font-family: serif;
7 word-spacing: 0.5pt;
8 text-indent: 0pt;
9}
10
11A:visited {
12 color: #204080;
13}
14A:link {
15 color: #102040;
16}
17h1 {
18 color: #103060;
19 font-size: 150%;
20}
21
22h2 {
23 color: #103060;
24 font-size: 80%;
25}
26
27code, pre {
28 font-family: monospace;
29}
30
31pre {
32 color: #102040;
33}
34
35code {
36 color:#0060c0;
37}
38
39.example {
40 margin-right: 20px;
41 margin-left: 20px;
42
43 padding-left: 20px;
44 padding-right: 20px;
45 padding-top: 0px;
46 padding-bottom: 0px;
47
48 background-color: #808080;
49 border-style: solid;
50 border-width: 1px;
51 border-color: #000000;
52}
53
54.leftbdr {
55 font-family: sans-serif;
56 color: #103060;
57 background-color: #606060;
58 border-style: solid;
59 border-width: 1px;
60 border-color: #000000;
61}
62.leftbody A:visited {
63 color: #102040;
64 text-decoration: none;
65}
66.leftbody A:link {
67 color: #102040;
68 text-decoration: none;
69}
diff --git a/xdelta3/www/xdelta3.html b/xdelta3/www/xdelta3.html
deleted file mode 100755
index 3bddfd9..0000000
--- a/xdelta3/www/xdelta3.html
+++ /dev/null
@@ -1,89 +0,0 @@
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2<html>
3<head>
4 <title>Xdelta3 delta compression library (BETA)</title>
5 <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
6 <link rel="stylesheet" type="text/css" href="xdelta3.css"/>
7</head>
8<body>
9
10<!-- $Format: "$WWWLeftNavBar$" $ --!>
11<table cellpadding="20px" width=700> <tr> <td class="leftbdr" valign=top height=600 width=100> <div class="leftbody"> <h1>Xdelta</h1> <a href="xdelta3.html">overview</a><br> <a href="xdelta3-cmdline.html">command&nbsp;line</a><br> <a href="xdelta3-api-guide.html">api&nbsp;guide</a><br> <br><a href="http://xdelta.org">xdelta.org</a></h2> </div> </td> <td valign=top width=500>
12
13<!-- Copyright (C) 2003 and onward. Joshua P. MacDonald --!>
14
15<h1>version three?</h1>
16
17Xdelta3 is the third and latest release of Xdelta, which is a set of tools and
18APIs for reading and writing compressed <em>deltas</em>. Deltas encode the
19differences between two versions of a document. This release features a
20completely new compression engine, several algorithmic improvements, a fully
21programmable interface modelled after zlib, in addition to a command-line
22utility, use of the RFC3284 (VCDIFF) encoding, a python extension, and now
2364-bit support.<p>
24
25Xdelta3 is <em>tiny</em>. A minimal, fully functional VCDIFF decoder library
26pipes in at 16KB. The command-line utility complete with encoder/decoder
27tools, external compression support, and the <code>djw</code> secondary
28compression routines, is just under 60KB, slightly larger than a
29<code>gzip</code> executable.<p>
30
31Xdelta3 has few dependencies because it's capable of stand-alone file
32compression (i.e., what zlib and gzip do). The stand-alone compression of
33Xdelta3/VCDIFF is 10-20% worse than <code>gzip</code>, you may view this as
34paying for the convenience-cost of having a single encoding, tool, and api
35designed to do both <em>data-compression</em> and <em>differencing</em> at
36once.<p>
37
38The Xdelta3 command-line tool, <code>xdelta3</code>, supports several
39convenience routines. Delta compression works when the two inputs are
40similar, but often we would like to compute the difference between two
41compressed documents. <code>xdelta3</code> has (optional) support to
42recognize externally compressed inputs and process them correctly. This
43support is facilitated, in part, using the VCDIFF <em>application header</em>
44field to store <code>xdelta3</code> meta-data, which includes the original
45file names (if any) and codes to incidate whether the inputs were externally
46compressed. Applications may provide their own application header.<p>
47
48<h1>what are version one and version two?</h1>
49
50Many shortcomings in the Xdelta1.x release are fixed in its replacement,
51Xdelta3. Xdelta1 used both a simplistic compression algorithm and a
52simplistic encoding. For example, Xdelta1 compresses the entire document at
53once and thus uses memory proportional to the input size.<p>
54
55The Xdelta1 compression engine made no attempt to find matching strings
56smaller than say 16 or 32 bytes, and the encoding does not attempt to
57efficiently encode the <code>COPY</code> and <code>ADD</code> instructions
58which constitute a delta. For documents with highly similar data, however,
59these techniques degrade performance by a relatively insignificant amount.
60(Xdelta1.x compresses the delta with Zlib to improve matters, but this
61dependency stinks.)<p>
62
63Despite leaving much to be desired, Xdelta1 showed that you can do well
64without great complexity; as it turns out, the particulars of the compression
65aengine are a relatively insignificant compared to the difficulty of
66programming an application that uses delta-compression. Better solve that
67first.<p>
68
69What we want are <em>systems</em> that manage compressed storage and network
70communication. The second major release, Xdelta2, addresses these issues.
71Xdelta2 features a storage interface -- part database and part file system --
72which allows indexing and labeling compressed documents. The feature set is
73similar to RCS. The Xdelta2 interface supports efficient algorithms for
74<em>extracting</em> deltas between any pair of versions in storage. The
75extraction technique also does not rely on hierarchy or centralizing the
76namespace, making the techniques ideal for peer-to-peer communication and
77proxy architectures. I am grateful to Mihut Ionescu for implementing the
78Xproxy HTTP delta-compressing proxy system based on this interface and
79studying the benefits of delta-compression in that context. Xdelta2 stressed
80the Xdelta1 compression engine beyond its limits; so Xdelta3 is designed as
81the ideal replacement. The Xdelta2 techniques are yet to be ported to the new
82implementation.<p>
83
84</td>
85</tr>
86</table>
87
88</body>
89</html>
diff --git a/xdelta3/xdelta3-main.h b/xdelta3/xdelta3-main.h
index 887f368..e2fe460 100755
--- a/xdelta3/xdelta3-main.h
+++ b/xdelta3/xdelta3-main.h
@@ -322,7 +322,7 @@ static int
322main_version (void) 322main_version (void)
323{ 323{
324 /* $Format: " P(RINT \"VERSION=3.$Xdelta3Version$\\n\");" $ */ 324 /* $Format: " P(RINT \"VERSION=3.$Xdelta3Version$\\n\");" $ */
325 P(RINT "VERSION=3.0m\n"); 325 P(RINT "VERSION=3.0n\n");
326 return EXIT_SUCCESS; 326 return EXIT_SUCCESS;
327} 327}
328 328
diff --git a/xdelta3/xdelta3-regtest.py b/xdelta3/xdelta3-regtest.py
index 03d3a2e..9e90f8b 100755
--- a/xdelta3/xdelta3-regtest.py
+++ b/xdelta3/xdelta3-regtest.py
@@ -31,8 +31,6 @@ MIN_SIZE = 0
31 31
32TIME_TOO_SHORT = 0.050 32TIME_TOO_SHORT = 0.050
33 33
34MIN_REPS = 1
35MAX_REPS = 1
36SKIP_TRIALS = 2 34SKIP_TRIALS = 2
37MIN_TRIALS = 3 35MIN_TRIALS = 3
38MAX_TRIALS = 15 36MAX_TRIALS = 15
@@ -208,13 +206,12 @@ class RcsFile:
208 self.Vstr(v), 206 self.Vstr(v),
209 self.Verf(v+1), 207 self.Verf(v+1),
210 self.Vstr(v+1))) 208 self.Vstr(v+1)))
211 print 'testing %s %s: ideal %.3f%%: time %.7f: in %u/%u trials' % \ 209 print 'testing %s %s: ideal %.3f%%: time %.7f: in %u trials' % \
212 (os.path.basename(self.fname), 210 (os.path.basename(self.fname),
213 self.Vstr(v+1), 211 self.Vstr(v+1),
214 result.r1.ideal, 212 result.r1.ideal,
215 result.time.mean, 213 result.time.mean,
216 result.trials, 214 result.trials)
217 result.reps)
218 ntrials.append(result) 215 ntrials.append(result)
219 216
220 os.remove(self.Verf(self.totrev-1)) 217 os.remove(self.Verf(self.totrev-1))
@@ -316,15 +313,14 @@ class Bucks:
316# 313#
317# 314#
318class TimeRun: 315class TimeRun:
319 def __init__(self,runnable,set_reps=1,reps=MIN_REPS,max_reps=MAX_REPS,\ 316 def __init__(self,runnable,
320 skip_trials=SKIP_TRIALS,min_trials=MIN_TRIALS,max_trials=MAX_TRIALS, \ 317 skip_trials=SKIP_TRIALS,min_trials=MIN_TRIALS,max_trials=MAX_TRIALS,
321 min_stddev_pct=MIN_STDDEV_PCT): 318 min_stddev_pct=MIN_STDDEV_PCT):
322 319
323 min_trials = min(min_trials,max_trials) 320 min_trials = min(min_trials,max_trials)
324 self.trials = 0 321 self.trials = 0
325 self.measured = [] 322 self.measured = []
326 self.r1 = None 323 self.r1 = None
327 self.reps = reps
328 while 1: 324 while 1:
329 try: 325 try:
330 os.remove(DFILE) 326 os.remove(DFILE)
@@ -335,7 +331,7 @@ class TimeRun:
335 start_time = time.time() 331 start_time = time.time()
336 start_clock = time.clock() 332 start_clock = time.clock()
337 333
338 result = runnable.Run(self.trials, self.reps) 334 result = runnable.Run(self.trials)
339 335
340 if self.r1 == None: 336 if self.r1 == None:
341 self.r1 = result 337 self.r1 = result
@@ -343,15 +339,8 @@ class TimeRun:
343 total_clock = (time.clock() - start_clock) 339 total_clock = (time.clock() - start_clock)
344 total_time = (time.time() - start_time) 340 total_time = (time.time() - start_time)
345 341
346 elap_time = max((total_time) / self.reps, 0.000001) 342 elap_time = max((total_time), 0.000001)
347 elap_clock = max((total_clock) / self.reps, 0.000001) 343 elap_clock = max((total_clock), 0.000001)
348
349 #print 'trial: %d' % self.trials
350 if set_reps and runnable.canrep and total_time < TIME_TOO_SHORT and self.reps < max_reps:
351 self.reps = max(self.reps+1,int(self.reps * TIME_TOO_SHORT / total_time))
352 self.reps = min(self.reps,max_reps)
353 #print 'continue: need more reps: %d' % self.reps
354 continue
355 344
356 self.trials = self.trials + 1 345 self.trials = self.trials + 1
357 346
@@ -478,29 +467,37 @@ class Xdelta3Pair:
478 self.newv = newv 467 self.newv = newv
479 return self 468 return self
480 469
481 def Run(self,trial,reps): 470 def Run(self,trial):
482 RunXdelta3(['-P', 471
483 '%d' % reps] + 472 # TODO: move '-S djw' somewhere else
484 self.extra + 473 encode_args = self.extra + \
485 [self.encode_args, 474 [ '-S', 'djw' ] + \
486 self.presrc, 475 [self.encode_args,
487 self.old, 476 self.presrc,
488 self.new, 477 self.old,
489 DFILE]) 478 self.new,
490 if trial > 0: 479 DFILE]
491 return None 480
492 self.dinfo = Xdelta3Info(self.new,DFILE) 481 decode_args = [self.decode_args,
493 if self.dinfo.extcomp: 482 self.presrc,
494 raise SkipRcsException('ext comp') 483 self.old,
495 RunXdelta3([self.decode_args, 484 DFILE,
496 self.presrc, 485 RFILE]
497 self.old, 486 try:
498 DFILE, 487 RunXdelta3(encode_args)
499 RFILE]) 488 if trial > 0:
500 RunCommand(('cmp', 489 return None
501 self.new, 490 self.dinfo = Xdelta3Info(self.new,DFILE)
502 RFILE)) 491 if self.dinfo.extcomp:
503 return self.dinfo 492 raise SkipRcsException('ext comp')
493 RunXdelta3(decode_args)
494 RunCommand(('cmp',
495 self.new,
496 RFILE))
497 return self.dinfo
498 except CommandError:
499 print 'encode args: %s' % ' '.join(encode_args)
500 print 'decode args: %s' % ' '.join(decode_args)
504 501
505def Test(): 502def Test():
506 rcsf = RcsFinder() 503 rcsf = RcsFinder()
@@ -529,15 +526,10 @@ def Decimals(max):
529 return l 526 return l
530 527
531class Xdelta3Run1: 528class Xdelta3Run1:
532 def __init__(self,file,reps=0): 529 def __init__(self,file):
533 self.file = file 530 self.file = file
534 self.reps = reps 531 def Run(self,trial):
535 self.canrep = 1 532 RunXdelta3(['-efq', self.file, DFILE])
536 def Run(self,trial,reps):
537 if self.reps:
538 assert(reps == 1)
539 reps = self.reps
540 RunXdelta3(['-P', '%d' % reps, '-efq', self.file, DFILE])
541 if trial > 0: 533 if trial > 0:
542 return None 534 return None
543 return Xdelta3Info(self.file,DFILE) 535 return Xdelta3Info(self.file,DFILE)
@@ -546,8 +538,7 @@ class GzipRun1:
546 def __init__(self,file): 538 def __init__(self,file):
547 self.file = file 539 self.file = file
548 self.canrep = 0 540 self.canrep = 0
549 def Run(self,trial,reps): 541 def Run(self,trial):
550 assert(reps == 1)
551 RunCommandIO(['gzip', '-cf'], self.file, DFILE) 542 RunCommandIO(['gzip', '-cf'], self.file, DFILE)
552 if trial > 0: 543 if trial > 0:
553 return None 544 return None
@@ -560,8 +551,8 @@ def SetFileSize(F,L):
560 os.close(fd) 551 os.close(fd)
561 552
562def ReportSpeed(L,tr,desc): 553def ReportSpeed(L,tr,desc):
563 print '%s 0-run length %u: dsize %u: time %.3f ms: encode %.0f B/sec: in %ux%u trials' % \ 554 print '%s 0-run length %u: dsize %u: time %.3f ms: encode %.0f B/sec: in %u trials' % \
564 (desc, L, tr.r1.dsize, tr.time.mean * 1000.0, ((L+tr.r1.dsize) / tr.time.mean), tr.trials, tr.reps) 555 (desc, L, tr.r1.dsize, tr.time.mean * 1000.0, ((L+tr.r1.dsize) / tr.time.mean), tr.trials)
565 556
566def MakeBigFiles(rcsf): 557def MakeBigFiles(rcsf):
567 rand = random.Random() 558 rand = random.Random()
@@ -586,39 +577,6 @@ def MakeBigFiles(rcsf):
586 return (TMPDIR + "/big.1", 577 return (TMPDIR + "/big.1",
587 TMPDIR + "/big.2") 578 TMPDIR + "/big.2")
588 579
589def BigFileRun(f1, f2):
590
591 testcases = [
592 # large_look large_step small_look small_chain small_lchain
593 # ssmatch try_lazy max_lazy long_enough promote
594# ['-DC', '10,1,4,36,13,0,1,512,256,0'],
595# ['-DC', '10,1,4,36,13,0,1,256,128,0'],
596# ['-DC', '10,1,4,36,13,0,1,128,64,0'],
597# ['-DC', '10,1,4,36,13,0,1,64,32,0'],
598
599 ['-DC', '11,7,4,2,1,0,0,110,178,1'],
600 ['-DC', '11,7,4,3,1,0,0,110,178,1'],
601 ['-DC', '11,7,4,4,1,0,0,110,178,1'],
602 ['-DC', '11,7,4,5,1,0,0,110,178,1'],
603 ['-DC', '11,7,4,2,1,0,0,110,100,1'],
604 ['-DC', '11,7,4,2,1,0,0,110,50,1'],
605 ['-DC', '11,7,4,2,1,0,0,110,25,1'],
606 ]
607
608 for test in testcases:
609 runner = Xdelta3Pair()
610 runner.extra = test
611 result = TimeRun(runner.Runner(f1, 1, f2, 2))
612
613 print 'test %s dsize %d: time %.7f: in %u/%u trials' % \
614 (test,
615 result.r1.dsize,
616 result.time.mean,
617 result.trials,
618 result.reps)
619 #end
620 return 1
621
622class RandomTestResult: 580class RandomTestResult:
623 def __init__(self, round, config, runtime, compsize): 581 def __init__(self, round, config, runtime, compsize):
624 self.round = round 582 self.round = round
@@ -672,15 +630,15 @@ class RandomTester:
672 def RandomConfig(self): 630 def RandomConfig(self):
673 631
674 input_ranges = [ 632 input_ranges = [
675 (7, 9, 12, 'large_look'), 633 (9, 9, 9, 'large_look'),
676 (1, 9, 17, 'large_step'), 634 (1, 4.5, 8, 'large_step'),
677 (4, 4, 4, 'small_look'), # Note: disabled 635 (4, 4, 4, 'small_look'), # Note: disabled
678 (1, 10, 100, 'small_chain'), 636 (1, 10, 30, 'small_chain'),
679 (1, 5, 50, 'small_lchain'), 637 (1, 3.5, 6, 'small_lchain'),
680 (0, 0, 0, 'ssmatch'), # Note: disabled 638 (0, 0, 0, 'ssmatch'), # Note: disabled
681 (1, 1, 1, 'trylazy'), # Note: enabled 639 (1, 1, 1, 'trylazy'), # Note: enabled
682 (1, 128, 1024, 'max_lazy'), 640 (1, 128, 256, 'max_lazy'),
683 (1, 256, 2048, 'long_enough'), 641 (1, 256, 512, 'long_enough'),
684 (0, 0, 0, 'promote'), # Note: disabled 642 (0, 0, 0, 'promote'), # Note: disabled
685 ] 643 ]
686 644
@@ -697,7 +655,7 @@ class RandomTester:
697 else: 655 else:
698 val = -1 656 val = -1
699 while val < minv or val > maxv: 657 while val < minv or val > maxv:
700 val = int(self.random.expovariate(1.0 / mean)) 658 val = int(self.random.expovariate(1.0 / mean) + 0.5)
701 #end 659 #end
702 #end 660 #end
703 661
@@ -758,10 +716,14 @@ class RandomTester:
758 sized = [] 716 sized = []
759 717
760 for test in self.results: 718 for test in self.results:
719
720 # This scores ellipse has x-major (time) and y-minor (size)
761 ntime = (test.time()) / float(maxt) 721 ntime = (test.time()) / float(maxt)
762 nsize = (test.size()) / float(maxs) 722 nsize = (test.size()) / float(maxs)
763 score = math.sqrt((maxs / mins) * ntime * ntime + 723
764 (maxt / mint) * nsize * nsize) 724 wntime = ntime * (maxs / mins)
725 wnsize = nsize * (maxt / mint)
726 score = math.sqrt(wntime * wntime + wnsize * wnsize)
765 scored.append((score, test)) 727 scored.append((score, test))
766 timed.append((test.time(), test, score)) 728 timed.append((test.time(), test, score))
767 sized.append((test.size(), test, score)) 729 sized.append((test.size(), test, score))
diff --git a/xdelta3/xdelta3.prj b/xdelta3/xdelta3.prj
index 684b109..0740f58 100755
--- a/xdelta3/xdelta3.prj
+++ b/xdelta3/xdelta3.prj
@@ -1,44 +1,39 @@
1;; -*- Prcs -*- 1;; -*- Prcs -*-
2(Created-By-Prcs-Version 1 3 4) 2(Created-By-Prcs-Version 1 3 4)
3(Project-Description "") 3(Project-Description "")
4(Project-Version xdelta3 0 11) 4(Project-Version xdelta3 0 12)
5(Parent-Version xdelta3 0 10) 5(Parent-Version xdelta3 0 11)
6(Version-Log "compiles on cygwin, copyright years updated, python experiments") 6(Version-Log "fixes buffer-allocation in xdelta3-decode related to secondary, adds new -6 default compression level")
7(New-Version-Log "") 7(New-Version-Log "")
8(Checkin-Time "Wed, 31 Jan 2007 22:47:01 -0800") 8(Checkin-Time "Sun, 04 Feb 2007 01:49:07 -0800")
9(Checkin-Login jmacd) 9(Checkin-Login jmacd)
10(Populate-Ignore ()) 10(Populate-Ignore ())
11(Project-Keywords 11(Project-Keywords
12 (Xdelta3Version "0m") 12 (Xdelta3Version "0n")
13 (WWWLeftNavBar "<table cellpadding=\"20px\" width=700> <tr> <td class=\"leftbdr\" valign=top height=600 width=100> <div class=\"leftbody\"> <h1>Xdelta</h1> <a href=\"xdelta3.html\">overview</a><br> <a href=\"xdelta3-cmdline.html\">command&nbsp;line</a><br> <a href=\"xdelta3-api-guide.html\">api&nbsp;guide</a><br> <br><a href=\"http://xdelta.org\">xdelta.org</a></h2> </div> </td> <td valign=top width=500>")
14 ) 13 )
15(Files 14(Files
16 (COPYING (xdelta3/b/29_COPYING 1.1 644)) 15 (COPYING (xdelta3/b/29_COPYING 1.1 644))
17 16
18 (xdelta3-cfgs.h (xdelta3/9_xdelta3-cf 1.3 744)) 17 (xdelta3-cfgs.h (xdelta3/9_xdelta3-cf 1.4 744))
19 (xdelta3-decode.h (xdelta3/b/30_xdelta3-de 1.3 744)) 18 (xdelta3-decode.h (xdelta3/b/30_xdelta3-de 1.4 744))
20 (xdelta3-djw.h (xdelta3/8_xdelta3-dj 1.4 744)) 19 (xdelta3-djw.h (xdelta3/8_xdelta3-dj 1.5 744))
21 (xdelta3-fgk.h (xdelta3/7_xdelta3-fg 1.3 744)) 20 (xdelta3-fgk.h (xdelta3/7_xdelta3-fg 1.3 744))
22 (xdelta3-list.h (xdelta3/6_xdelta3-li 1.3 744)) 21 (xdelta3-list.h (xdelta3/6_xdelta3-li 1.3 644))
23 (xdelta3-main.h (xdelta3/5_xdelta3-ma 1.7 744)) 22 (xdelta3-main.h (xdelta3/5_xdelta3-ma 1.8 744))
24 (xdelta3-python.h (xdelta3/4_xdelta3-py 1.3 744)) 23 (xdelta3-python.h (xdelta3/4_xdelta3-py 1.3 644))
25 (xdelta3-regtest.py (xdelta3/10_xdelta3-re 1.6 744)) 24 (xdelta3-regtest.py (xdelta3/10_xdelta3-re 1.7 744))
26 (xdelta3-second.h (xdelta3/3_xdelta3-se 1.3 744)) 25 (xdelta3-second.h (xdelta3/3_xdelta3-se 1.4 744))
27 (xdelta3-test.h (xdelta3/2_xdelta3-te 1.5 744)) 26 (xdelta3-test.h (xdelta3/2_xdelta3-te 1.6 744))
28 (xdelta3.c (xdelta3/16_xdelta3.c 1.6 744)) 27 (xdelta3.c (xdelta3/16_xdelta3.c 1.7 744))
29 (xdelta3.h (xdelta3/1_xdelta3.h 1.5 744)) 28 (xdelta3.h (xdelta3/1_xdelta3.h 1.6 744))
30 29
31 (Makefile (xdelta3/0_Makefile 1.4 744)) 30 (Makefile (xdelta3/0_Makefile 1.5 744))
32 (setup.py (xdelta3/11_setup.py 1.2 744)) 31 (setup.py (xdelta3/11_setup.py 1.3 744))
33 32
34 (draft-korn-vcdiff.txt (xdelta3/b/22_draft-korn 1.1 644)) 33 (draft-korn-vcdiff.txt (xdelta3/b/22_draft-korn 1.1 644))
35 (www/xdelta3-api-guide.html (xdelta3/b/23_Xdelta3-ap 1.5 744))
36 (www/xdelta3-cmdline.html (xdelta3/b/25_xdelta3-cm 1.3 744))
37 (www/xdelta3.css (xdelta3/b/26_xdelta3.cs 1.3 744))
38 (www/xdelta3.html (xdelta3/b/24_Xdelta3.ht 1.5 744))
39 34
40 (badcopy.c (xdelta3/20_badcopy.c 1.2 744)) 35 (badcopy.c (xdelta3/20_badcopy.c 1.2 744))
41 (linkxd3lib.c (xdelta3/19_linkxd3lib 1.2 744)) 36 (linkxd3lib.c (xdelta3/19_linkxd3lib 1.3 744))
42 (rcs_junk.cc (xdelta3/15_rcs_junk.c 1.1 644)) 37 (rcs_junk.cc (xdelta3/15_rcs_junk.c 1.1 644))
43 38
44 (README (xdelta3/b/31_README 1.1 744)) 39 (README (xdelta3/b/31_README 1.1 744))
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-06 20:13:19 +0000