Skip to content

DRILL-8507, DRILL-8508 Better handling of partially missing parquet columns - #2937

Open
ychernysh wants to merge 3 commits into
apache:masterfrom
ychernysh:partially-missing-parquet-columns
Open

ychernysh wants to merge 3 commits into
apache:masterfrom
ychernysh:partially-missing-parquet-columns

Conversation

@ychernysh

@ychernysh ychernysh commented Aug 28, 2024 •

Copy link
Copy Markdown

DRILL-8507: Missing parquet columns quoted with backticks conflict with existing ones

DRILL-8508: Choosing the best suitable major type for a partially missing parquet column

Description

This PR aims to solve 2 separate Drill jiras at once: DRILL-8507 and DRILL-8508. They were reported separately because they solve different issues, but the coupling here is that DRILL-8508 depends on DRILL-8507 and requires it to be fixed first. But since in terms of code they share the same place, I decided to bring the commits in a single PR so it would be easier to review the code and understand the underlying motivations. Please review the 3 commits separately in order.

DRILL-8507 Missing parquet columns quoted with backticks conflict with existing ones
In this commit I replace col.toExpr() call with col.getAsUnescapedPath() in ParquetSchema#createMissingColumn() to get rid of some confusions when comparing quoted and unquoted field name. If the names itself are identical, the comparing should return true, but the difference in quotes make it return false. Obviously it is a bug.
However, the decision to use col.Expr() was made consciously based on the comment above that line. But I didn't really understand the motivation to do so... If anyone from the community can explain it, I would appreciate that.
Such change leads to a few regressions seen in unit tests. We should probably adjust the tests (because they relied on a bug), but I'm still not sure how exactly to do this properly. Here is how and why I've done it in the commit:

  1. TestCaseNullableTypes: removed testCoalesceWithUntypedNullValues* test methods because now we will always return Nullable Int for missing columns instead of Untyped Null. Other way to make these tests pass is to use CAST to cast the missing column to varchar, but it doesn't really make a lot of sense since based on the method name it expects exactly the Untyped Null
  2. TestParquetFilterPushDown: simply use CONVERT_TO to workaround NumberFormatException (because, again, now we have Nullable Int instead of Untyped Null)
  3. TestUntypedNull: moved testCoalesceOnNotExistentColumns* test methods to a separate TestParquetMissingColumns.java class and made them expect Nullable Int (for that same reason as above)
  4. DeltaQueriesTest (contrib/format-deltalake): some tests fail here after the fix because they use specific implicit columns, not listed here. There is no solution for these tests in this PR and we should probably handle it as a separate task (because it is probably something wrong with delta lake plugin)

TestParquetPartiallyMissingColumns is a new test class that brings unit tests for this exact fix. Check them out to understand what is the problem with backticking missing column. Also note that each of the following commits bring its own test cases in that class.

DRILL-8508 Choosing the best suitable major type for a partially missing parquet column (minor type solution)
This one is pretty simple: take that GroupScan table schema (that is merged from all of the parquet files to read) and pick the minor type for missing column from there

DRILL-8508 Choosing the best suitable major type for a partially missing parquet column (data mode solution)
Here I added the logic to enforce OPTIONAL data mode on ALL readers if we found a column missing or OPTIONAL. We use the same schema from the previous commit to catch such cases.
Note that I added synchronization in Metadata to properly resolve data modes (always choose least restrictive). I have tested a performance after that and didn't notice any significant impact.

Documentation

Since these changes only make Drill handle the cases that it was not able to handle before without any user interaction, there is no documentation needed.

Testing

Note that all of the 3 commits bring their own test cases in TestParquetPartiallyMissingColumns file. I added parquet data files in test resources for them.