Apache Iceberg managed tables

Apache Iceberg managed tables (formerly BigLake tables for Apache Iceberg in BigQuery) provide the foundation for building open-format lakehouses on Google Cloud. Iceberg managed tables offer the same fully managed experience as standard BigQuery tables, but store data in customer-owned storage buckets. Iceberg managed tables support the open Iceberg table format for better interoperability with open-source and third-party compute engines on a single copy of data.

Iceberg managed tables support the following features:

Architecture

Iceberg managed tables bring the convenience of BigQuery resource management to tables that reside in your own cloud buckets. You can use BigQuery and open-source compute engines on these tables without moving the data out of the buckets that you control. You must configure a Cloud Storage bucket before you start using Iceberg managed tables.

Using Iceberg managed tables has the following implications on your bucket:

  • BigQuery creates new data files in the bucket in response to write requests and background storage optimizations, such as DML statements and streaming.
  • Automatic compaction and clustering are performed on the data files in the bucket. After the expiration of the time travel window, data files are garbage collected. However, if the table is deleted, the associated data files aren't garbage collected. For more information, see Storage optimization.

Creating an Iceberg managed table is similar to creating BigQuery tables. Because it stores data in open formats on Cloud Storage, you must do the following:

  • Specify the Cloud resource connection with WITH CONNECTION to configure the connection credentials for BigQuery to access Cloud Storage.
  • Specify the file format of data storage as PARQUET with the file_format = PARQUET statement.
  • Specify the open-source metadata table format as ICEBERG with the table_format = ICEBERG statement.

Best practices

Directly changing or adding files to the bucket outside of BigQuery can lead to data loss or unrecoverable errors. The following table describes possible scenarios:

Operation Consequences Prevention
Add new files to the bucket outside BigQuery. Data loss: New files or objects added outside of BigQuery are not tracked by BigQuery. Untracked files are deleted by background garbage collection processes. Add data exclusively through BigQuery. This lets BigQuery track the files and prevent them from being garbage collected.
To prevent accidental additions and data loss, we also recommend restricting external tool write permissions on buckets containing Iceberg managed tables.
Create a new Iceberg managed table in a non-empty prefix. Data loss: Extant data isn't tracked by BigQuery, so these files are considered untracked, and deleted by background garbage collection processes. Only create new Iceberg managed tables in empty prefixes.
Modify or replace Iceberg managed table data files. Data loss: On external modification or replacement, the table fails a consistency check and becomes unreadable. Queries against the table fail.
There is no self-serve way to recover from this point. Contact support for data recovery assistance.		 Modify data exclusively through BigQuery. This lets BigQuery track the files and prevent them from being garbage collected.
To prevent accidental additions and data loss, we also recommend restricting external tool write permissions on buckets containing Iceberg managed tables.
Create two Iceberg managed tables on the same or overlapping URIs. Data loss: BigQuery doesn't bridge identical URI instances of Iceberg managed tables. Background garbage collection processes for each table will consider the opposite table's files as untracked, and delete them, causing data loss. Use unique URIs for each Iceberg managed table.

Cloud Storage bucket configuration best practices

The configuration of your Cloud Storage bucket and its connection with BigQuery have a direct impact on the performance, cost, data integrity, security, and governance of your Iceberg managed tables. The following are best practices to help with this configuration:

  • Select a name that clearly indicates that the bucket is only meant for Iceberg managed tables.

  • Choose single-region Cloud Storage buckets that are co-located in the same region as your BigQuery dataset. This coordination improves performance and lowers costs by avoiding data transfer charges.

  • By default, Cloud Storage stores data in the Standard storage class, which provides sufficient performance. To optimize data storage costs, you can enable Autoclass to automatically manage storage class transitions. Autoclass starts with the Standard storage class and moves objects that aren't accessed to progressively colder classes in order to reduce storage costs. When the object is read again, it's moved back to the Standard class.

  • Enable uniform bucket-level access and public access prevention.

  • Verify that the required roles are assigned to the correct users and service accounts.

  • To prevent accidental data deletion or corruption in your Cloud Storage bucket, restrict write and delete permissions for most users in your organization. You can do this by setting a bucket permission policy with conditions that deny PUT and DELETE requests for all users, except those that you specify.

  • Apply google-managed or customer-managed encryption keys for extra protection of sensitive data.

  • Enable audit logging for operational transparency, troubleshooting, and monitoring data access.

  • Keep the default soft delete policy (7 day retention) to protect against accidental deletions. However, if you find that data has been deleted, engage with support rather than restoring objects manually, as objects that are added or modified outside of BigQuery aren't tracked by BigQuery metadata.

  • Adaptive file sizing, automatic clustering, and garbage collection are enabled automatically and help with optimizing file performance and cost.

  • Avoid the following Cloud Storage features, as they are unsupported for Iceberg managed tables: