| 2 |
- |
1 |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
|
|
2 |
<html>
|
|
|
3 |
<head>
|
|
|
4 |
<title>Using and writing Ghostscript testing scripts</title>
|
|
|
5 |
<!-- $Id: Testing.htm,v 1.37 2005/10/20 19:46:23 ray Exp $ -->
|
|
|
6 |
<link rel="stylesheet" type="text/css" href="gs.css" title="Ghostscript Style">
|
|
|
7 |
</head>
|
|
|
8 |
|
|
|
9 |
<body>
|
|
|
10 |
<!-- [1.0 begin visible header] ============================================ -->
|
|
|
11 |
|
|
|
12 |
<!-- [1.1 begin headline] ================================================== -->
|
|
|
13 |
|
|
|
14 |
<h1>Using and writing Ghostscript testing scripts</h1>
|
|
|
15 |
|
|
|
16 |
<!-- [1.1 end headline] ==================================================== -->
|
|
|
17 |
|
|
|
18 |
<!-- [1.2 begin table of contents] ========================================= -->
|
|
|
19 |
|
|
|
20 |
<!-- [1.2 end table of contents] =========================================== -->
|
|
|
21 |
|
|
|
22 |
<h2>Table of contents</h2>
|
|
|
23 |
|
|
|
24 |
<blockquote><ul>
|
|
|
25 |
<li><a href="#General_overview">General overview</a>
|
|
|
26 |
<li><a href="#Running_tests">Running tests</a>
|
|
|
27 |
<ul>
|
|
|
28 |
<li><a href="#Individual_tests">Individual tests</a>
|
|
|
29 |
<li><a href="#Regression_testing">Regression testing</a>
|
|
|
30 |
</ul>
|
|
|
31 |
<li><a href="#Writing_new_tests">Writing new tests</a>
|
|
|
32 |
</ul></blockquote>
|
|
|
33 |
|
|
|
34 |
<!-- [1.3 begin hint] ====================================================== -->
|
|
|
35 |
|
|
|
36 |
<p>
|
|
|
37 |
This document describes how to use the scripts located in the <a
|
|
|
38 |
href="../toolbin/tests">toolbin/tests</a> directory, and conventions for
|
|
|
39 |
writing new testing scripts.
|
|
|
40 |
|
|
|
41 |
<p>For other information, see the <a href="Readme.htm">Ghostscript
|
|
|
42 |
overview</a> and the documentation related to <a
|
|
|
43 |
href="Maintain.htm">maintaining Ghostscript</a>.
|
|
|
44 |
|
|
|
45 |
<!-- [1.3 end hint] ======================================================== -->
|
|
|
46 |
|
|
|
47 |
<hr>
|
|
|
48 |
|
|
|
49 |
<!-- [1.0 end visible header] ============================================== -->
|
|
|
50 |
|
|
|
51 |
<!-- [2.0 begin contents] ================================================== -->
|
|
|
52 |
|
|
|
53 |
<h2><a name="General_overview"></a>General overview</h2>
|
|
|
54 |
|
|
|
55 |
<p>
|
|
|
56 |
The test scripts discussed here are written in Python, a language whose
|
|
|
57 |
implementation is freely available from <a class="offsite"
|
|
|
58 |
href="http://www.python.org"><tt>http://www.python.org</tt></a>. The
|
|
|
59 |
scripts require Python version 2.1 or later.
|
|
|
60 |
|
|
|
61 |
<h2><a name="Running_tests"></a>Running tests</h2>
|
|
|
62 |
|
|
|
63 |
<p>
|
|
|
64 |
On Unix and Linux systems, test scripts written in Python can be executed
|
|
|
65 |
directly simply by typing their name into the shell, e.g.,
|
|
|
66 |
|
|
|
67 |
<blockquote><pre>
|
|
|
68 |
toolbin/tests/check_source.py
|
|
|
69 |
</pre></blockquote>
|
|
|
70 |
|
|
|
71 |
<p>
|
|
|
72 |
On other systems, it may be necessary to invoke Python explicitly, e.g.,
|
|
|
73 |
|
|
|
74 |
<blockquote><pre>
|
|
|
75 |
python toolbin/tests/check_source.py
|
|
|
76 |
</pre></blockquote>
|
|
|
77 |
|
|
|
78 |
<p>
|
|
|
79 |
The test scripts will print information about any failures that occur.
|
|
|
80 |
|
|
|
81 |
<h3><a name="Individual_tests"></a>Individual tests</h3>
|
|
|
82 |
|
|
|
83 |
<p>
|
|
|
84 |
The individual test scripts are named
|
|
|
85 |
<b><tt>toolbin/tests/check_</tt></b><em>xxx</em><b><tt>.py</tt></b> (if they
|
|
|
86 |
do not run Ghostscript) or
|
|
|
87 |
<b><tt>toolbin/tests/gscheck_</tt></b><em>xxx</em><b><tt>.py</tt></b> (if
|
|
|
88 |
they do run Ghostscript). Any script with such a name can be run
|
|
|
89 |
individually, and is also normally run as part of regression testing
|
|
|
90 |
(described in the next section).
|
|
|
91 |
|
|
|
92 |
<p>
|
|
|
93 |
We don't list the individual test scripts here, because any such
|
|
|
94 |
documentation would inevitably be out of date most of the time. Each of
|
|
|
95 |
these scripts contains documentation about what it tests: we suggest you
|
|
|
96 |
read the documentation in the scripts.
|
|
|
97 |
|
|
|
98 |
<h3><a name="Regression_testing"></a>Regression testing</h3>
|
|
|
99 |
|
|
|
100 |
<p>
|
|
|
101 |
We run a nightly regression test to discover any obvious problems caused by
|
|
|
102 |
code checked in the previous day. Here is a list of the scripts and
|
|
|
103 |
supporting files that make up the regression test.
|
|
|
104 |
|
|
|
105 |
<h4>Top-level scripts</h4>
|
|
|
106 |
|
|
|
107 |
<dl>
|
|
|
108 |
<dl>
|
|
|
109 |
<dt><a href="../toolbin/tests/dump_testdb"><tt>dump_testdb</tt></a>
|
|
|
110 |
<dd>A debugging script that will print the contents of the database defined
|
|
|
111 |
by gsconf.testdatadb.
|
|
|
112 |
</dl>
|
|
|
113 |
|
|
|
114 |
<dl>
|
|
|
115 |
<dt><a href="../toolbin/tests/make_testdb"><tt>make_testdb</tt></a>
|
|
|
116 |
<dd>This script creates an initial test database. It uses gsconf.baselinegs
|
|
|
117 |
to create raster data from the test files and computes their MD5 sums and
|
|
|
118 |
stores them in the gsconf.testdatadb database.
|
|
|
119 |
</dl>
|
|
|
120 |
|
|
|
121 |
<dl>
|
|
|
122 |
<dt><a href="../toolbin/tests/make_two_versions"><tt>make_two_versions</tt></a>
|
|
|
123 |
<dd>A helper script to make two versions of a particular file for visual
|
|
|
124 |
diffing or manual analysis. When a test fails nightly regression, this is
|
|
|
125 |
generally the first investigative step.
|
|
|
126 |
</dl>
|
|
|
127 |
|
|
|
128 |
<dl>
|
|
|
129 |
<dt><a href="../toolbin/tests/make_two_pdfversions"><tt>make_two_pdfversions</tt></a>
|
|
|
130 |
<dd>Same as above, except for pdfwrite regressions.
|
|
|
131 |
</dl>
|
|
|
132 |
|
|
|
133 |
<dl>
|
|
|
134 |
<dt><a href="../toolbin/tests/testdiff"><tt>testdiff</tt></a>
|
|
|
135 |
<dd>this script provides the difference between two sets of regression results.
|
|
|
136 |
if end date is omitted, the current date will be used.
|
|
|
137 |
</dl>
|
|
|
138 |
|
|
|
139 |
<dl>
|
|
|
140 |
<dt><a href="../toolbin/tests/revert_baseline"><tt>revert_baseline</tt></a>
|
|
|
141 |
<dd>In cases where a baseline has been accidentally updated, this script
|
|
|
142 |
will revert the database entry to the MD5 sum computed with
|
|
|
143 |
gsconf.baselinegs.
|
|
|
144 |
</dl>
|
|
|
145 |
|
|
|
146 |
<dl>
|
|
|
147 |
<dt><a href="../toolbin/tests/revert_pdfbaseline"><tt>revert_pdfbaseline</tt></a>
|
|
|
148 |
<dd>Same as above, except for pdfwrite baselines.
|
|
|
149 |
</dl>
|
|
|
150 |
|
|
|
151 |
<dl>
|
|
|
152 |
<dt><a href="../toolbin/tests/run_nightly"><tt>run_nightly</tt></a>
|
|
|
153 |
<dd>This is the control script (usually invoked by cron) that controls the
|
|
|
154 |
nightly test run. It's responsible for checking the latest code out of CVS,
|
|
|
155 |
building a new Ghostscript to compare with and launching the test suite via
|
|
|
156 |
run_regression.
|
|
|
157 |
</dl>
|
|
|
158 |
|
|
|
159 |
<dl>
|
|
|
160 |
<dt><a href="../toolbin/tests/run_regression"><tt>run_regression</tt></a>
|
|
|
161 |
<dd>This script runs the full gamut of regression tests using files from
|
|
|
162 |
gsconf.comparedir. It differentiates files by extension and controls what
|
|
|
163 |
tests get run and with what options.
|
|
|
164 |
</dl>
|
|
|
165 |
|
|
|
166 |
<dl>
|
|
|
167 |
<dt><a href="../toolbin/tests/update_baseline"><tt>update_baseline</tt></a>
|
|
|
168 |
<dd>This script is invoked to update the MD5 sum in the test database when a
|
|
|
169 |
nightly regression is really a progression. Generally after noticing that
|
|
|
170 |
the output from make_two_versions is acceptable or better, this script is
|
|
|
171 |
run to log the changes to the database.
|
|
|
172 |
</dl>
|
|
|
173 |
|
|
|
174 |
<dl>
|
|
|
175 |
<dt><a href="../toolbin/tests/update_pdfbaseline"><tt>update_pdfbaseline</tt></a>
|
|
|
176 |
<dd>Same as above, except for pdfwrite baselines.
|
|
|
177 |
</dl>
|
|
|
178 |
|
|
|
179 |
<dl>
|
|
|
180 |
<dt><a href="../toolbin/tests/update_specific"><tt>update_specific</tt></a> <tt><em>cvs-date-spec</em></tt>
|
|
|
181 |
<dd>Update a series of baselines to a specific datestamp; accepts a series of "flag filename"
|
|
|
182 |
lines on stdin, where flag is either N or P to specify whether the normal or pdfwrite baselines
|
|
|
183 |
should be updated.
|
|
|
184 |
<dl>
|
|
|
185 |
|
|
|
186 |
<dt><a href="../toolbin/tests/get_baselines"><tt>get_baselines</tt></a>
|
|
|
187 |
<dd>This script returns the change log for the test database starting
|
|
|
188 |
at a given date.
|
|
|
189 |
</dl>
|
|
|
190 |
</dl>
|
|
|
191 |
|
|
|
192 |
<h4>Support files</h4>
|
|
|
193 |
|
|
|
194 |
<dl>
|
|
|
195 |
|
|
|
196 |
<dl>
|
|
|
197 |
<dt><a href="../toolbin/tests/fuzzy.c"><tt>fuzzy.c</tt></a>
|
|
|
198 |
<dd>A fuzzy comparison tool appropriate for tests where exact binary matches
|
|
|
199 |
aren't appropriate.
|
|
|
200 |
</dl>
|
|
|
201 |
|
|
|
202 |
<dl>
|
|
|
203 |
<dt><a href="../toolbin/tests/cmpi.py"><tt>cmpi.py</tt></a>
|
|
|
204 |
<dd>An image comparison tool for exploring the differences between two
|
|
|
205 |
raster files. It can switch between the two, display only the differences
|
|
|
206 |
and localize, highlight and jump between areas where differences occur.
|
|
|
207 |
This can be a useful way of examining regression difference to decide
|
|
|
208 |
if they are progressions or regressions. Requires python imaging and
|
|
|
209 |
python Tkinter.
|
|
|
210 |
</dl>
|
|
|
211 |
|
|
|
212 |
<dl>
|
|
|
213 |
<dt><a href="../toolbin/tests/gsconf.py"><tt>gsconf.py</tt></a>
|
|
|
214 |
<dd>This is the configuration framework for the scripts above. It reads
|
|
|
215 |
configuration files and makes available those configuration options to
|
|
|
216 |
the rest of the testing framework.
|
|
|
217 |
</dl>
|
|
|
218 |
|
|
|
219 |
<dl>
|
|
|
220 |
<dt><a href="../toolbin/tests/testing.cfg.example"><tt>testing.cfg.example</tt></a>
|
|
|
221 |
<dd>This is an example configuration file for the scripts above. It controls
|
|
|
222 |
where files are found, where Ghostscript executables are and the location of
|
|
|
223 |
the test database. Most test configuration will be in this file. It must
|
|
|
224 |
be copied to testing.cfg in order for the tests to find it.
|
|
|
225 |
</dl>
|
|
|
226 |
|
|
|
227 |
<dl>
|
|
|
228 |
<dt><a href="../toolbin/tests/testing.cfg"><tt>testing.cfg</tt></a>
|
|
|
229 |
<dd>This is the configuration file for the scripts above. It controls
|
|
|
230 |
where files are found, where Ghostscript executables are and the location of
|
|
|
231 |
the test database. Most test configuration will be in this file. The file
|
|
|
232 |
testing.cfg.example above can be used as a template.
|
|
|
233 |
</dl>
|
|
|
234 |
|
|
|
235 |
<dl>
|
|
|
236 |
<dt><a href="../toolbin/tests/gstestgs.py"><tt>gstestgs.py</tt></a>
|
|
|
237 |
<dd>This provides classes for running tests that actually execute
|
|
|
238 |
Ghostcript.
|
|
|
239 |
</dl>
|
|
|
240 |
|
|
|
241 |
<dl>
|
|
|
242 |
<dt><a href="../toolbin/tests/gsparamsets.py"><tt>gsparamsets.py</tt></a>
|
|
|
243 |
<dd>What parameters tests get run with by default are stored in this file.
|
|
|
244 |
Between this configuration information and the information in gsconf.py all
|
|
|
245 |
configurable testing parameters should be covered.
|
|
|
246 |
</dl>
|
|
|
247 |
|
|
|
248 |
<dl>
|
|
|
249 |
<dt><a href="../toolbin/tests/gsutil.py"><tt>gsutil.py</tt></a>
|
|
|
250 |
<dd>Various utility routines used by the regression test scripts.
|
|
|
251 |
</dl>
|
|
|
252 |
|
|
|
253 |
<dl>
|
|
|
254 |
<dt><a href="../toolbin/tests/gssum.py"><tt>gssum.py</tt></a>
|
|
|
255 |
<dd>Helper functions that compute, compare and store MD5 sums.
|
|
|
256 |
</dl>
|
|
|
257 |
|
|
|
258 |
<dl>
|
|
|
259 |
<dt><a href="../toolbin/tests/rasterdb.py"><tt>rasterdb.py</tt></a>
|
|
|
260 |
<dd>Hepler functions for accessing the raster database.
|
|
|
261 |
</dl>
|
|
|
262 |
|
|
|
263 |
</dl>
|
|
|
264 |
|
|
|
265 |
<h2><a name="Writing_new_tests"></a>Writing new tests</h2>
|
|
|
266 |
|
|
|
267 |
<p>
|
|
|
268 |
Some of Ghostscript's test scripts follow a set of conventions that allow
|
|
|
269 |
them to be run either stand-alone or as part of a suite; in particular, they
|
|
|
270 |
can be run as part of the nightly regression test suite. In this section,
|
|
|
271 |
we provide pointers to documentation on how to write new tests that follow
|
|
|
272 |
these conventions, since that will make them the most useful.
|
|
|
273 |
|
|
|
274 |
<p>
|
|
|
275 |
The test scripts are based on Python's <b><tt>unittest</tt></b> module. We
|
|
|
276 |
suggest that if you are not familiar with this module, you read the
|
|
|
277 |
documentation, which is available at <a class="offsite"
|
|
|
278 |
href="http://www.python.org/doc/current/lib/module-unittest.html">http://www.python.org/doc/current/lib/module-unittest.html</a>.
|
|
|
279 |
|
|
|
280 |
<p>
|
|
|
281 |
Ghostscript specializes the <b><tt>unittest</tt></b> module by defining
|
|
|
282 |
subclasses, which all individual tests use in place of those in
|
|
|
283 |
<b><tt>unittest</tt></b>. These subclasses are defined in <a
|
|
|
284 |
href="../toolbin/tests/gstestutils.py">toolbin/tests/gstestutils.py</a>.
|
|
|
285 |
|
|
|
286 |
<p>
|
|
|
287 |
Since code documentation separate from the code itself is always out of
|
|
|
288 |
date, we have decided to maintain the primary documentation for writing new
|
|
|
289 |
tests in <a href="../toolbin/tests/gstestutils.py">gstestutils.py</a> rather
|
|
|
290 |
than here in a separate document. Please read that file for more
|
|
|
291 |
information.
|
|
|
292 |
|
|
|
293 |
<!-- [2.0 end contents] ==================================================== -->
|
|
|
294 |
|
|
|
295 |
<!-- [3.0 begin visible trailer] =========================================== -->
|
|
|
296 |
<hr>
|
|
|
297 |
|
|
|
298 |
<p>
|
|
|
299 |
<small>Copyright © 2002 artofcode LLC. All rights reserved.</small>
|
|
|
300 |
|
|
|
301 |
<p>
|
|
|
302 |
<small>This file is part of AFPL Ghostscript. See the <a
|
|
|
303 |
href="Public.htm">Aladdin Free Public License</a> (the "License") for full
|
|
|
304 |
details of the terms of using, copying, modifying, and redistributing AFPL
|
|
|
305 |
Ghostscript.</small>
|
|
|
306 |
|
|
|
307 |
<p>
|
|
|
308 |
<small>Ghostscript version 8.53, 20 October 2005
|
|
|
309 |
|
|
|
310 |
<!-- [3.0 end visible trailer] ============================================= -->
|
|
|
311 |
|
|
|
312 |
</body>
|
|
|
313 |
</html>
|