Extended Protocol Packet Metadata

Warning

SingleStore 9.0 gives you the opportunity to preview, evaluate, and provide feedback on new and upcoming features prior to their general availability. In the interim, SingleStore 8.9 is recommended for production workloads, which can later be upgraded to SingleStore 9.0.

SingleStore uses the same format of column definition packet when communicating between the client and the server as the MySQL protocol. By default, packet metadata sent to the client consists of 12 bytes, which does not allow distinguishing non-MySQL protocol-compatible data types. Extending the packet metadata size allows the client to distinguish between data types that are not compatible with MySQL, such as BSON and VECTOR. Refer to enum_field_types Reference for a list of MySQL supported data types.

By default, SingleStore does not send extended metadata in the protocol packet for backward compatibility. When extended metadata in the protocol packet is disabled (default behavior), BSON and VECTOR type columns are sent to the clients as LONGBLOB and VARCHAR, respectively. Refer to Data Types for more information on the respective data types.

Extended protocol packet metadata is supported in SingleStore version 8.5.28 and later.

Note: The MariaDB C library does not support extended metadata format.

Enable Extended Metadata

To enable sending extended metadata in the protocol packet for BSON and VECTOR data types:

SQL

SET SESSION enable_extended_types_metadata = TRUE;

enable_extended_types_metadata is a session variable that can be set globally. SingleStore recommends configuring this engine variable for each session.

Extended Metadata Format

The packet metadata for extended types has the following structure:

Type	Name	Description
`string<lenenc>`	catalog	Catalog name. Currently, this value is always `"def"`.
`string<lenenc>`	database	Database name
`string<lenenc>`	table	Virtual table name
`string<lenenc>`	org_table	Physical table name
`string<lenenc>`	name	Virtual column name
`string<lenenc>`	org_name	Physical column name
`int<lenenc>`		Length of the fixed length fields. This value is greater than or equal to 12 (0x0C) bytes.
`int<2>`	character_set	The column character set.
`int<4>`	column_length	Maximum length of the field in bytes.
`int<1>`	type	Column type.
`int<2>`	flags	Column definition flags.
`int<1>`	decimals	Number of decimal places.
`int<2>`	unused	Two unused zero-bytes.
`int<1>`	type code of extended datatype	Type code of the extended type. This value is `2` for `VECTOR` data types and `1` for `BSON` types.

where, lenenc represents a length-encoded string.

For VECTOR types, the protocol packet metadata contains the following additional bytes:

Type	Name	Description
`int<4>`	number of vector elements	Dimension of the `VECTOR` type.
`int<1>`	type of vector elements	The type of elements in the vector data.

For `VECTOR` Types

For VECTOR types, the protocol packet metadata contains the following 6 bytes in addition to the default 12 bytes:

1 byte: Specifies the type code. This value is 2 for VECTOR data types.
4 bytes: Specifies the dimension (number of elements) of the vector data.

1 byte: Specifies the type of elements in the vector data. It can have the following values based on the type of the vector elements:

Value	Vector Element Type
`1`	`F32`
`2`	`F64`
`3`	`I8`
`4`	`I16`
`5`	`I32`
`6`	`I64`

For example, the protocol packet metadata for a VECTOR(23) type column (starting from the "Length of the fixed length field" byte) is:

Extended Types Disabled	Extended Types Enabled
0c 3f 00 ff ff ff ff fd 90 00 00 00 00	12 3f 00 ff ff ff ff fd 90 00 00 00 00 02 17 00 00 00 01

where the extended bytes represent the following:

02: Type code for VECTOR type.
17 00 00 00: Dimension (length) of the vector data, i.e., 23.
01: Vector element type, i.e., F32.

For `BSON` Types

For BSON types, the protocol packet metadata contains 1 byte in addition to the default 12 bytes. This additional byte specifies the type code, which is 1 for BSON types.

Extended Protocol Packet Metadata

On this page

Enable Extended Metadata

Extended Metadata Format

For `VECTOR` Types

For `BSON` Types

Was this article helpful?

On this page

Was this article helpful?

Extended Protocol Packet Metadata

On this page

Enable Extended Metadata

Extended Metadata Format

For VECTOR Types

For BSON Types

Was this article helpful?

On this page

Was this article helpful?

For `VECTOR` Types

For `BSON` Types