You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Encode long Huffman values without growing-integer shifts - #304
HuffmanEncoder.encode() appends each symbol to a single Python integer (final_num <<= length; final_num |= code). Every step copies the whole integer, so encoding is quadratic in the value length, and long values such as cookies dominate encode time.
For bytes values of 256 bytes or more, this builds a small map from each distinct input byte to its code written as a string of 0/1 digits, joins those strings, pads with EOS (1) bits to a whole byte and converts once with int(bits, 2).to_bytes(...). The map is rebuilt on every call from the encoder's own huffman_code_list / huffman_code_list_lengths, so nothing is cached between calls and changes to those tables are still picked up.
Everything else keeps the existing loop: values shorter than 256 bytes, bytes subclasses, HuffmanEncoder subclasses, and code tables that are not plain lists of ints or contain a code with a length outside 1..30 or a value that does not fit its length (the loop's masking/overlap behaviour for such tables is not reproduced, so they fall back). The output is identical in all cases.
HuffmanEncoder.encode()appends each symbol to a single Python integer (final_num <<= length; final_num |= code). Every step copies the whole integer, so encoding is quadratic in the value length, and long values such as cookies dominate encode time.For
bytesvalues of 256 bytes or more, this builds a small map from each distinct input byte to its code written as a string of0/1digits, joins those strings, pads with EOS (1) bits to a whole byte and converts once withint(bits, 2).to_bytes(...). The map is rebuilt on every call from the encoder's ownhuffman_code_list/huffman_code_list_lengths, so nothing is cached between calls and changes to those tables are still picked up.Everything else keeps the existing loop: values shorter than 256 bytes,
bytessubclasses,HuffmanEncodersubclasses, and code tables that are not plain lists of ints or contain a code with a length outside 1..30 or a value that does not fit its length (the loop's masking/overlap behaviour for such tables is not reproduced, so they fall back). The output is identical in all cases.Benchmarks