Skip to content

Core: Accept Arrow PyCapsule exporters in append and overwrite - #4460

Open
Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:feat/2680-arrow-capsule-input
Open

Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:feat/2680-arrow-capsule-input

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor

Rationale for this change

Writes rejected anything that was not a pyarrow.Table or pyarrow.RecordBatchReader, which forced users of Polars, arro3, and other Arrow implementations to import PyArrow themselves and convert first.

Accept objects that implement the Arrow PyCapsule stream interface (__arrow_c_stream__) in append and overwrite, and import them as a PyArrow table. RecordBatchReader keeps the existing streaming path. The error message now names the accepted types.

Are these changes tested?

Yes. A wrapper implementing only __arrow_c_stream__ is written through both append and overwrite, and an object without the interface is rejected. tests/table passes.

Are there any user-facing changes?

Yes. append and overwrite accept any object implementing the Arrow PyCapsule stream interface.

Writes rejected anything that was not a pyarrow Table or RecordBatchReader,
which forced users of Polars, arro3, and other Arrow implementations to
import pyarrow themselves and convert first.

Accept objects that implement the Arrow PyCapsule stream interface and
import them as a pyarrow Table. RecordBatchReader keeps the streaming
path, and the error message names the accepted types.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant