Proposed Lightning Talk: Regression testing for layout fidelity with convwatch and lenient-pdf-compare

LibreOffice, and OpenOffice.org before that, have used a variety of tools under the name convwatch to perform regression testing by comparing rendered documents (usually printed through a generic postscript driver and then converted to png or jped) against some reference. Convwatch takes care of printing the file and delegates image conversion and comparison to GhostScript and ImageMagick. In the past, the process has not only been hampered by stability issues during the conversion process, but also by the fact that rendering is not completely platform independent: font anti-aliasing and kerning work slightly different on different platforms, resulting in differences on the pixel level that usually don't affect the perceivable quality of the printed document, but will lead to false positives when comparing documents on a platform other than that where the reference has been created. This issue can be addressed by reducing the resolution or allowing for some threshold of allowed differences when comparing the complete page image. Unfortunately, this can also leave degradations in rendering left unnoticed.

We have developed a tool that is able to compare PDF documents both structurally and visually. Instead of allowing for a global threshold of differences, lenient-pdf-compare is able to weigh local differences and thus ignore certain changes (caused by different anti-aliasing) without reducing the overall sensitivity to changes. lenient-pdf-compare is also able to visualise differences per page as png file.

We're currently publishing lenient-pdf-compare on github.com under the LGPLv3 and intend to integrate it into the current incarnation of convwatch, hopefully making convwatch more useful in the future.

lenient-pdf-compare is developed by Nico Hezel, a student of information technology and media, currently employed by E&E information consultants AG, developers of ESCRIBA. ESCRIBA is an enterprise document generation solution which uses OpenOffice.org/LibreOffice as editor and rendering service. E&E is using lenient-pdf-compare as a regression testing tool for ESCRIBA, and is publishing lenient-pdf-compare as part of an effort contribute to the OpenOffice/LibreOffice community.

In the lightning talk (15 minutes), we want to show a simple convwatch session using lenient-pdf-compare to detect differences between documents.

Speakers:

Nico Hezel, developer of lenient-pdf-compare

Peter Jentsch, development lead for ESCRIBA and LibreOffice contributor (flat ODF support, libxslt port of the LibreOffice xslt filter).