Skip to content

DRILL-8511: Overflow appeared when the batch reached rows limit - #2943

Draft
rymarm wants to merge 2 commits into
apache:masterfrom
rymarm:DRILL-8511
Draft

rymarm wants to merge 2 commits into
apache:masterfrom
rymarm:DRILL-8511

Conversation

@rymarm

@rymarm rymarm commented Sep 18, 2024

Copy link
Copy Markdown
Member

DRILL-8511: Overflow appeared when the batch reached rows limit

Description

The size-aware scan framework fails to end the batch.

Framework tries to reallocate the vector on batch end, due to a hidden, minor bug in BitColumnWriter - which in general is not notable, but in a specific case, when the initial vector allocation size limit is exceeded and a reader reaches the batch row size limit.

BitColumnWriter uses instead of a write index a value count and this causes unexpected vector reallocation (look at the changes).

Documentation

No changes required.

Testing

Manual tests

@rymarm
rymarm marked this pull request as draft September 19, 2024 13:29
@paul-rogers

Copy link
Copy Markdown
Contributor

@rymarm, thanks for finding this issue. I believe your fix is correct. The prepareWrite() call exists to fill any remaining bits with 0s (i.e. "fill empties."). The number to be filled is, indeed, valueCount - 1. So, I think this is a good fix.

I would feel more confident if we had a unit test. That this bug exists suggests that, despite the rather excessive number of tests I wrote way back when, I somehow omitted this case. Perhaps a reason is that the bit vector is special: it uses code different than the other vectors, and so needs its own tests. Tests for the fixed-width vectors are in TestFixedWidthWriter and TestFillEmpties. Perhaps you can whip up a TestBitVector that does a few tests, including for the special case you uncovered. In particular, it would be good to make sure that the following cases work:

  • Write some values to the bit vector to simulate the first few records setting the bits.
  • Claim that the batch has ended and the batch wrote a bunch of other records without writing to the bit vector.