Paimon

An HMS or Glue table is treated as Paimon when its properties contain table_type with value PAIMON, case-insensitively. Tables from a Paimon FileSystem catalog are always Paimon. Lakelet loads the latest Paimon schema from the table location.

Data Types

Same as paimon-rust.

Create schema

CREATE SCHEMA [IF NOT EXISTS] schema_name;

Drop schema

DROP SCHEMA [IF EXISTS] schema_name [CASCADE | RESTRICT];

Create table

CREATE TABLE [IF NOT EXISTS] table_name (
    column_definition [, ...]
    [, PRIMARY KEY (column_name [, ...])]
)
[PARTITIONED BY (column_name [, ...])]
[USING PAIMON]
[WITH ('key' = 'value' [, ...])];

column_definition:
    column_name data_type
    [NULL | NOT NULL]
    [COMMENT 'text']
    [PRIMARY KEY]

• IF NOT EXISTS leaves an existing table unchanged; it does not compare or update its definition. Without it, creating an existing table is an error.
• Columns are nullable by default. Primary-key columns become non-nullable and cannot be declared explicitly NULL. Only one primary key is allowed, either inline or as a table constraint.
• PARTITIONED BY references existing column names without types.
• USING PAIMON is optional. Other table formats are not accepted.
• WITH accepts single-quoted string keys and values.
• Table locations are chosen under the configured warehouse.

Table options

Set Paimon table options with WITH ('key' = 'value' [, ...]) when creating a table. Keys and values, including numbers and booleans, must be single-quoted strings. The options below follow the Paimon table options reference.

Bucket Configuration

Merge Engine

For primary-key tables, partial-update keeps the latest non-null value of each field. Set 'ignore-delete' = 'true' or 'partial-update.ignore-delete' = 'true' to ignore retract rows; otherwise they are rejected. Once enabled, ignore-delete cannot be changed back to false. Creating partial-update tables with sequence groups or aggregation options is not supported.

The basic aggregation engine supports fixed buckets and dynamic buckets ('bucket' = '-1') when the primary key includes all partition columns. Configure functions with fields.<field>.aggregate-function or fields.default-aggregate-function. Supported functions are sum, product, min, max, last_value, first_value, last_non_null_value, first_non_null_value, bool_and, bool_or, and listagg. Sequence fields always use last_value and cannot have an explicit aggregate function. Retract rows, deletion-vector writes, cross-partition dynamic bucket writes, and advanced aggregation options such as ignore-retract, distinct, nested-key, count-limit, and sequence groups are not supported.

For deletion-vector-enabled primary-key tables using deduplicate, batch reads hide uncompacted level-0 files by default. Set 'deletion-vectors.merge-on-read' = 'true' to include and merge them on read. This option only takes effect when 'deletion-vectors.enabled' = 'true' and only affects batch snapshot reads. Deletion-vector writes and merge-on-read are not supported for partial-update or aggregation tables.

Global Index Options

Global index creation in Paimon requires row tracking and data evolution; primary-key tables and tables with deletion vectors enabled are not supported. Setting these options alone does not build an index.

global-index.vindex.read-thread-num is independent of global-index.thread-num. Set both explicitly if index search tasks and Vindex file reads should use the same concurrency limit.

Variant Shredding Options

These options apply to VARIANT columns written to Parquet. They control the file layout while preserving the logical VARIANT type. Configured shredding takes precedence over inferred shredding. Tables without VARIANT columns, or without shredding enabled, use the normal file layout.

Other Options

Cross-partition update mode applies when 'bucket' = '-1' and the primary key does not contain every partition column; it has no separate option.

Drop table

DROP TABLE [IF EXISTS] table_name;

Writing tables

Existing Paimon tables support INSERT INTO and INSERT OVERWRITE. The storage credentials must allow writes to the table location.

INSERT INTO table_name [(column_name [, ...])]
{ VALUES (expression [, ...]) [, ...] | query };

Braces group alternatives; choose either VALUES or a query. Create the destination table before inserting rows or query results. Creation and insertion are separate operations; an INSERT failure does not undo table creation. INSERT results report the number of rows written.

Overwrite data

INSERT OVERWRITE table_name [(column_name [, ...])]
{ VALUES (expression [, ...]) [, ...] | query };

For an unpartitioned table, INSERT OVERWRITE replaces all its data. For a partitioned table, it replaces only the partitions present in the new data. Other partitions remain unchanged.

Metadata Table

Metadata tables expose snapshots, files, options, and other table information. Append the metadata table name to the table name with a $ separator:

SELECT * FROM `table_name$snapshots`;

Note that Lakelet names the list of data files data_files, consistent with the other table formats; Paimon's native files spelling is not accepted.

data_files

Returns one row for each data file in the latest snapshot:

SELECT * FROM `table_name$data_files`;

partitions

Returns one row for each partition:

SELECT * FROM `table_name$partitions`;

snapshots

Returns one row for each snapshot of the table:

SELECT * FROM `table_name$snapshots`;

manifests

Returns one row for each manifest file of the latest snapshot:

SELECT * FROM `table_name$manifests`;

options

Returns the table options that are set explicitly, one row per option:

SELECT * FROM `table_name$options`;

schemas

Returns one row for each schema version of the table:

SELECT * FROM `table_name$schemas`;

tags

Returns one row for each tag of the table:

SELECT * FROM `table_name$tags`;

branches

Returns one row for each branch of the table:

SELECT * FROM `table_name$branches`;

table_indexes

Returns one row for each committed global index file, including BTree indexes, vector indexes, and deletion-vector metadata:

SELECT * FROM `table_name$table_indexes`;

physical_files_size

Returns a single row with the physical disk usage of the table, categorized by manifest, data, and index files:

SELECT * FROM `table_name$physical_files_size`;

aggregation_fields

Returns one row per table field, including its field-specific options and comment:

SELECT * FROM `table_name$aggregation_fields`;

Fields without these options have [] in the two option columns. For example, an option 'fields.total.aggregate-function' = 'sum' appears as [sum] in function and [fields.total.aggregate-function] in function_options.

audit_log

Exposes data rows with a leading rowkind column. This describes row kinds in the data, not a history of SQL statements, users, or access events. A normal query reads the current table state; it is not a complete history of changes.

SELECT * FROM `table_name$audit_log` LIMIT 10;

The row kinds returned depend on the table's data and read behavior. If the table enables table-read.sequence-number.enabled, an additional _SEQUENCE_NUMBER column follows rowkind.

buckets

Returns file statistics grouped by partition and bucket in the current snapshot:

SELECT * FROM `table_name$buckets`;

consumers

Returns the consumer progress recorded for the table. Querying this metadata table does not create or advance a consumer.

SELECT * FROM `table_name$consumers`;

referenced_files_size

Returns file counts and sizes aggregated across all referenced snapshots, including branches and tags, one row per source:

SELECT * FROM `table_name$referenced_files_size`;