Benchmarks and Speed
| Author: | Stefan Behnel |
|---|
lxml.etree is a very fast XML library. Most of this is due to the speed of libxml2, e.g. the parser and serialiser, or the XPath engine. Other areas of lxml were specifically written for high performance in high-level operations, such as the tree iterators.
On the other hand, the simplicity of lxml sometimes hides internal operations that are more costly than the API suggests. If you are not aware of these cases, lxml may not always perform as you expect. A common example in the Python world is the Python list type. New users often expect it to be a linked list, while it actually is implemented as an array, which results in a completely different complexity for common operations.
Similarly, the tree model of libxml2 is more complex than what lxml's ElementTree API projects into Python space, so some operations may show unexpected performance. Rest assured that most lxml users will not notice this in real life, as lxml is very fast in absolute numbers. It is definitely fast enough for most applications, so lxml is probably somewhere between 'fast enough' and 'the best choice' for yours. Read some messages from happy users to see what we mean.
This text describes where lxml.etree (abbreviated to 'lxe') excels, gives hints on some performance traps and compares the overall performance to the original ElementTree (ET) and cElementTree (cET) libraries by Fredrik Lundh. The cElementTree library is a fast C-implementation of the original ElementTree.
Contents
General notes
First thing to say: there is an overhead involved in having a DOM-like C library mimic the ElementTree API. As opposed to ElementTree, lxml has to generate Python representations of tree nodes on the fly when asked for them, and the internal tree structure of libxml2 results in a higher maintenance overhead than the simpler top-down structure of ElementTree. What this means is: the more of your code runs in Python, the less you can benefit from the speed of lxml and libxml2. Note, however, that this is true for most performance critical Python applications. No one would implement Fourier transformations in pure Python when you can use NumPy.
The up side then is that lxml provides powerful tools like tree iterators, XPath and XSLT, that can handle complex operations at the speed of C. Their pythonic API in lxml makes them so flexible that most applications can easily benefit from them.
How to read the timings
The statements made here are backed by the (micro-)benchmark scripts bench_etree.py, bench_xpath.py and bench_objectify.py that come with the lxml source distribution. They are distributed under the same BSD license as lxml itself, and the lxml project would like to promote them as a general benchmarking suite for all ElementTree implementations. New benchmarks are very easy to add as tiny test methods, so if you write a performance test for a specific part of the API yourself, please consider sending it to the lxml mailing list.
The timings presented below compare lxml 4.6.3 (with libxml2 2.9.10) to the latest released versions of ElementTree (with cElementTree as accelerator module) in the standard library of CPython 3.8.10. They were run single-threaded on a 2.3GHz 64bit double core Intel i5 machine under Ubuntu Linux 20.04 (Focal).
The scripts run a number of simple tests on the different libraries, using different XML tree configurations: different tree sizes (T1-4), with or without attributes (-/A), with or without ASCII string or unicode text (-/S/U), and either against a tree or its serialised XML form (T/X). In the result extracts cited below, T1 refers to a 3-level tree with many children at the third level, T2 is swapped around to have many children below the root element, T3 is a deep tree with few children at each level and T4 is a small tree, slightly broader than deep. If repetition is involved, this usually means running the benchmark in a loop over all children of the tree root, otherwise, the operation is run on the root node (C/R).
As an example, the character code (SATR T1) states that the benchmark was running for tree T1, with plain string text (S) and attributes (A). It was run against the root element (R) in the tree structure of the data (T).
Note that very small operations are repeated in integer loops to make them measurable. It is therefore not always possible to compare the absolute timings of, say, a single access benchmark (which usually loops) and a 'get all in one step' benchmark, which already takes enough time to be measurable and is therefore measured as is. An example is the index access to a single child, which cannot be compared to the timings for getchildren(). Take a look at the concrete benchmarks in the scripts to understand how the numbers compare.
Parsing and Serialising
Serialisation is an area where lxml excels. The reason is that it executes entirely at the C level, without any interaction with Python code. The results are rather impressive, especially for UTF-8, which is native to libxml2. While 20 to 40 times faster than (c)ElementTree 1.2 (which was part of the standard library before Python 2.7/3.2), lxml is still several times faster than the much improved ElementTree 1.3 in recent Python versions:
lxe: tostring_utf16 (S-TR T1) 5.9340 msec/pass cET: tostring_utf16 (S-TR T1) 38.3270 msec/pass lxe: tostring_utf16 (UATR T1) 6.2032 msec/pass cET: tostring_utf16 (UATR T1) 37.7944 msec/pass lxe: tostring_utf16 (S-TR T2) 6.1841 msec/pass cET: tostring_utf16 (S-TR T2) 40.2577 msec/pass lxe: tostring_utf8 (S-TR T2) 4.6697 msec/pass cET: tostring_utf8 (S-TR T2) 30.5173 msec/pass lxe: tostring_utf8 (U-TR T3) 1.2085 msec/pass cET: tostring_utf8 (U-TR T3) 9.0246 msec/pass
The difference is somewhat smaller for plain text serialisation:
lxe: tostring_text_ascii (S-TR T1) 2.6727 msec/pass cET: tostring_text_ascii (S-TR T1) 2.9683 msec/pass lxe: tostring_text_ascii (S-TR T3) 0.6952 msec/pass cET: tostring_text_ascii (S-TR T3) 1.0073 msec/pass lxe: tostring_text_utf16 (S-TR T1) 2.7366 msec/pass cET: tostring_text_utf16 (S-TR T1) 7.3647 msec/pass lxe: tostring_text_utf16 (U-TR T1) 3.0322 msec/pass cET: tostring_text_utf16 (U-TR T1) 7.5922 msec/pass
The tostring() function also supports serialisation to a Python unicode string object, which is currently faster in ElementTree under CPython 3.8:
lxe: tostring_text_unicode (S-TR T1) 2.7645 msec/pass cET: tostring_text_unicode (S-TR T1) 1.1806 msec/pass lxe: tostring_text_unicode (U-TR T1) 2.9871 msec/pass cET: tostring_text_unicode (U-TR T1) 1.1659 msec/pass lxe: tostring_text_unicode (S-TR T3) 0.7446 msec/pass cET: tostring_text_unicode (S-TR T3) 0.4532 msec/pass lxe: tostring_text_unicode (U-TR T4) 0.0048 msec/pass cET: tostring_text_unicode (U-TR T4) 0.0134 msec/pass
For parsing, lxml.etree and cElementTree compete for the medal. Depending on the input, either of the two can be faster. The (c)ET libraries use a very thin layer on top of the expat parser, which is known to be very fast. Here are some timings from the benchmarking suite:
lxe: parse_bytesIO (SAXR T1) 14.2074 msec/pass cET: parse_bytesIO (SAXR T1) 7.9336 msec/pass lxe: parse_bytesIO (S-XR T3) 1.4477 msec/pass cET: parse_bytesIO (S-XR T3) 2.1925 msec/pass lxe: parse_bytesIO (UAXR T3) 8.4128 msec/pass cET: parse_bytesIO (UAXR T3) 12.2926 msec/pass
And another couple of timings from a benchmark that Fredrik Lundh used to promote cElementTree, comparing a number of different parsers. First, parsing a 274KB XML file containing Shakespeare's Hamlet:
xml.etree.ElementTree.parse done in 0.006 seconds xml.etree.cElementTree.parse done in 0.007 seconds xml.etree.cElementTree.XMLParser.feed(): 6636 nodes read in 0.006 seconds lxml.etree.parse done in 0.004 seconds drop_whitespace.parse done in 0.004 seconds lxml.etree.XMLParser.feed(): 6636 nodes read in 0.004 seconds minidom tree read in 0.066 seconds
And a 3.4MB XML file containing the Old Testament:
xml.etree.ElementTree.parse done in 0.037 seconds xml.etree.cElementTree.parse done in 0.036 seconds xml.etree.cElementTree.XMLParser.feed(): 25317 nodes read in 0.036 seconds lxml.etree.parse done in 0.025 seconds drop_whitespace.parse done in 0.022 seconds lxml.etree.XMLParser.feed(): 25317 nodes read in 0.026 seconds minidom tree read in 0.194 seconds
Here are the same benchmarks again, but including the memory usage of the process in KB before and after parsing (using os.fork() to make sure we start from a clean state each time). For the 274KB hamlet.xml file:
Memory usage: 9256 xml.etree.ElementTree.parse done in 0.006 seconds Memory usage: 12764 (+3508) xml.etree.cElementTree.parse done in 0.007 seconds Memory usage: 12764 (+3508) xml.etree.cElementTree.XMLParser.feed(): 6636 nodes read in 0.006 seconds Memory usage: 12720 (+3464) lxml.etree.parse done in 0.004 seconds Memory usage: 15052 (+5796) drop_whitespace.parse done in 0.004 seconds Memory usage: 14040 (+4784) lxml.etree.XMLParser.feed(): 6636 nodes read in 0.004 seconds Memory usage: 15812 (+6556) minidom tree read in 0.066 seconds Memory usage: 15332 (+6076)
And for the 3.4MB Old Testament XML file:
Memory usage: 12456 xml.etree.ElementTree.parse done in 0.037 seconds Memory usage: 23288 (+10832) xml.etree.cElementTree.parse done in 0.036 seconds Memory usage: 23288 (+10832) xml.etree.cElementTree.XMLParser.feed(): 25317 nodes read in 0.036 seconds Memory usage: 23644 (+11220) lxml.etree.parse done in 0.025 seconds Memory usage: 31404 (+18948) drop_whitespace.parse done in 0.022 seconds Memory usage: 28752 (+16296) lxml.etree.XMLParser.feed(): 25317 nodes read in 0.026 seconds Memory usage: 33924 (+21500) minidom tree read in 0.194 seconds Memory usage: 31284 (+18828)
As can be seen from the sizes, both lxml.etree and cElementTree are rather memory friendly and fast. Comparing to older CPython versions, the memory footprint of the minidom library was considerably reduced in CPython 3.3, by about a factor of 4 in this case.
For plain parser performance, lxml.etree and cElementTree tend to stay rather close to each other, usually within a factor of two, with winners well distributed over both sides. Similar timings can be observed for the iterparse() function:
lxe: iterparse_bytesIO (SAXR T1) 20.3598 msec/pass cET: iterparse_bytesIO (SAXR T1) 10.8948 msec/pass lxe: iterparse_bytesIO (UAXR T3) 10.1640 msec/pass cET: iterparse_bytesIO (UAXR T3) 12.9926 msec/pass
However, if you benchmark the complete round-trip of a serialise-parse cycle, the numbers will look similar to these:
lxe: write_utf8_parse_bytesIO (S-TR T1) 18.9857 msec/pass cET: write_utf8_parse_bytesIO (S-TR T1) 35.7475 msec/pass lxe: write_utf8_parse_bytesIO (UATR T2) 22.4853 msec/pass cET: write_utf8_parse_bytesIO (UATR T2) 42.6254 msec/pass lxe: write_utf8_parse_bytesIO (S-TR T3) 3.3801 msec/pass cET: write_utf8_parse_bytesIO (S-TR T3) 11.2493 msec/pass lxe: write_utf8_parse_bytesIO (SATR T4) 0.4263 msec/pass cET: write_utf8_parse_bytesIO (SATR T4) 1.0326 msec/pass
For applications that require a high parser throughput of large files, and that do little to no serialization, both cET and lxml.etree are a good choice. The cET library is particularly fast for iterparse applications that extract small amounts of data or aggregate information from large XML data sets that do not fit into memory. If it comes to round-trip performance, however, lxml is multiple times faster in total. So, whenever the input documents are not considerably larger than the output, lxml is the clear winner.
Regarding HTML parsing, Ian Bicking has done some benchmarking on lxml's HTML parser, comparing it to a number of other famous HTML parser tools for Python. lxml wins this contest by quite a length. To give an idea, the numbers suggest that lxml.html can run a couple of parse-serialise cycles in the time that other tools need for parsing alone. The comparison even shows some very favourable results regarding memory consumption.
Liza Daly has written an article that presents a couple of tweaks to get the most out of lxml's parser for very large XML documents. She quite favourably positions lxml.etree as a tool for high-performance XML parsing.
Finally, xml.com has a couple of publications about XML parser performance. Farwick and Hafner have written two interesting articles that compare the parser of libxml2 to some major Java based XML parsers. One deals with event-driven parser performance, the other one presents benchmark results comparing DOM parsers. Both comparisons suggest that libxml2's parser performance is largely superior to all commonly used Java parsers in almost all cases. Note that the C parser benchmark results are based on xmlbench, which uses a simpler setup for libxml2 than lxml does.
The ElementTree API
Since all three libraries implement the same API, their performance is easy to compare in this area. A major disadvantage for lxml's performance is the different tree model that underlies libxml2. It allows lxml to provide parent pointers for elements and full XPath support, but also increases the overhead of tree building and restructuring. This can be seen from the tree setup times of the benchmark (given in seconds):
lxe: -- S- U- -A SA UA
T1: 0.0219 0.0254 0.0257 0.0216 0.0259 0.0259
T2: 0.0234 0.0279 0.0283 0.0271 0.0318 0.0307
T3: 0.0051 0.0050 0.0058 0.0218 0.0233 0.0231
T4: 0.0001 0.0001 0.0001 0.0004 0.0004 0.0004
cET: -- S- U- -A SA UA
T1: 0.0035 0.0029 0.0078 0.0031 0.0031 0.0029
T2: 0.0047 0.0051 0.0053 0.0046 0.0055 0.0048
T3: 0.0016 0.0216 0.0027 0.0021 0.0023 0.0026
T4: 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
The timings are somewhat close to each other, although cET can be several times faster than lxml for larger trees. One of the reasons is that lxml must encode incoming string data and tag names into UTF-8, and additionally discard the created Python elements after their use, when they are no longer referenced. ElementTree represents the tree itself through these objects, which reduces the overhead in creating them.
Child access
The same tree overhead makes operations like collecting children as in list(element) more costly in lxml. Where cET can quickly create a shallow copy of their list of children, lxml has to create a Python object for each child and collect them in a list:
lxe: root_list_children (--TR T1) 0.0036 msec/pass cET: root_list_children (--TR T1) 0.0005 msec/pass lxe: root_list_children (--TR T2) 0.0634 msec/pass cET: root_list_children (--TR T2) 0.0086 msec/pass
This handicap is also visible when accessing single children:
lxe: first_child (--TR T2) 0.0601 msec/pass cET: first_child (--TR T2) 0.0548 msec/pass lxe: last_child (--TR T1) 0.0570 msec/pass cET: last_child (--TR T1) 0.0534 msec/pass
... unless you also add the time to find a child index in a bigger list. ET and cET use Python lists here, which are based on arrays. The data structure used by libxml2 is a linked tree, and thus, a linked list of children:
lxe: middle_child (--TR T1) 0.0892 msec/pass cET: middle_child (--TR T1) 0.0510 msec/pass lxe: middle_child (--TR T2) 2.3038 msec/pass cET: middle_child (--TR T2) 0.0508 msec/pass
Element creation
As opposed to ET, libxml2 has a notion of documents that each element must be in. This results in a major performance difference for creating independent Elements that end up in independently created documents:
lxe: create_elements (--TC T2) 0.8032 msec/pass cET: create_elements (--TC T2) 0.0675 msec/pass
Therefore, it is always preferable to create Elements for the document they are supposed to end up in, either as SubElements of an Element or using the explicit Element.makeelement() call:
lxe: makeelement (--TC T2) 0.8030 msec/pass cET: makeelement (--TC T2) 0.0625 msec/pass lxe: create_subelements (--TC T2) 0.8621 msec/pass cET: create_subelements (--TC T2) 0.0923 msec/pass
So, if the main performance bottleneck of an application is creating large XML trees in memory through calls to Element and SubElement, cET is the best choice. Note, however, that the serialisation performance may even out this advantage, especially for smaller trees and trees with many attributes.
Merging different sources
A critical action for lxml is moving elements between document contexts. It requires lxml to do recursive adaptations throughout the moved tree structure.
The following benchmark appends all root children of the second tree to the root of the first tree:
lxe: append_from_document (--TR T1,T2) 1.3800 msec/pass cET: append_from_document (--TR T1,T2) 0.0513 msec/pass lxe: append_from_document (--TR T3,T4) 0.0150 msec/pass cET: append_from_document (--TR T3,T4) 0.0026 msec/pass
Although these are fairly small numbers compared to parsing, this easily shows the different performance classes for lxml and (c)ET. Where the latter do not have to care about parent pointers and tree structures, lxml has to deep traverse the appended tree. The performance difference therefore increases with the size of the tree that is moved.
This difference is not always as visible, but applies to most parts of the API, like inserting newly created elements:
lxe: insert_from_document (--TR T1,T2) 5.2345 msec/pass cET: insert_from_document (--TR T1,T2) 0.0732 msec/pass
or replacing the child slice by a newly created element:
lxe: replace_children_element (--TC T1) 0.0720 msec/pass cET: replace_children_element (--TC T1) 0.0105 msec/pass
as opposed to replacing the slice with an existing element from the same document:
lxe: replace_children (--TC T1) 0.0060 msec/pass cET: replace_children (--TC T1) 0.0050 msec/pass
While these numbers are too small to provide a major performance impact in practice, you should keep this difference in mind when you merge very large trees. Note that Elements have a