Row Format

Overview

Apache Fory Row Format is a cache-friendly, random-access binary format designed for high-performance data processing. Unlike traditional serialization formats that require full deserialization, the row format enables:

• Random Field Access: Read individual fields without deserializing the entire row
• Zero-Copy Operations: Direct memory access without data transformation
• Cache-Friendly Layout: Optimized memory layout for CPU cache efficiency
• Cross-Language Support: Consistent binary format across Java, C++, and Python

Fory provides two row format variants:

Format	Languages	Use Case
Standard Format	Java, C++, Python	Cross-language compatibility
Compact Format	Java only	Space efficiency, smaller rows

Format Comparison

Feature	Standard Format	Compact Format
Field Slot Size	Fixed 8 bytes	Natural width (1, 2, 4, or 8 bytes)
Null Bitmap Size	8-byte aligned	Byte-aligned, can borrow padding
Null Bitmap Position	Before field slots	After field slots (at end)
Fixed-Size Structs	Variable region (offset+size)	Inline in fixed region
Field Ordering	Schema-defined order	Sorted by alignment
All Non-Nullable	Bitmap still present	Bitmap skipped entirely
Alignment	Strict 8-byte	Relaxed (2, 4, or 8-byte)

---

Standard Row Format

The standard format prioritizes cross-language compatibility and simplicity with uniform 8-byte field slots.

Design Principles

1. 8-Byte Alignment: All major structures are aligned to 8-byte boundaries for optimal memory access
2. Fixed-Width Field Slots: Every field uses an 8-byte slot for uniform offset calculation
3. Null Bitmap: Compact null tracking using bit vectors
4. Relative Offsets: Variable-length data uses relative offsets for sub-buffer navigation

Row Binary Layout

A row stores structured data with the following layout:

+----------------+------------------+------------------+-----+------------------+------------------+
|  Null Bitmap   |  Field 0 Slot    |  Field 1 Slot    | ... |  Field N-1 Slot  |  Variable Data   |
+----------------+------------------+------------------+-----+------------------+------------------+
|  B bytes       |     8 bytes      |     8 bytes      |     |     8 bytes      |  Variable size   |

Null Bitmap

The null bitmap tracks which fields contain null values:

• Size: ((num_fields + 63) / 64) * 8 bytes (rounded up to nearest 8-byte word)
• Encoding: Each bit corresponds to a field index
  • Bit value 1 = field is null
  • Bit value 0 = field is not null
• Bit Order: Bit 0 of the first byte corresponds to field 0

Example: For 10 fields, bitmap size = ((10 + 63) / 64) * 8 = 8 bytes

Field Slots

Each field occupies a fixed 8-byte slot regardless of its actual data type:

• Slot Offset: bitmap_size + field_index * 8
• Total Fixed Region: bitmap_size + num_fields * 8 bytes

Field Slot Contents by Type:

Type Category	Slot Contents
Fixed-width	Value stored directly (zero-padded)
Variable-width	Offset + Size encoding (see below)

Variable-Width Data Encoding

Variable-length fields (strings, arrays, maps, nested structs) store an offset-size pair in their slot:

+---------------------------+---------------------------+
|    Relative Offset        |         Size              |
|       (32 bits)           |       (32 bits)           |
+---------------------------+---------------------------+
|<-------------- 64-bit field slot value -------------->|

• Relative Offset (upper 32 bits): Offset from the row's base address
• Size (lower 32 bits): Size of the variable-width data in bytes

Encoding:

offset_and_size = (relative_offset << 32) | size

Decoding:

relative_offset = (offset_and_size >> 32) & 0xFFFFFFFF
size = offset_and_size & 0xFFFFFFFF

Variable Data Region

Variable-length data is stored after the fixed region:

• Data is written sequentially as fields are set
• Each variable-length value is padded to 8-byte alignment
• Padding bytes are zeroed for deterministic output

Array Binary Layout

Arrays store homogeneous sequences of elements:

+------------------+------------------+------------------+
|  Element Count   |   Null Bitmap    |   Element Data   |
+------------------+------------------+------------------+
|     8 bytes      |     B bytes      |   Variable size  |

Array Header

Field	Size	Description
Element Count	8 bytes	Number of elements (uint64)
Null Bitmap	((count + 63) / 64) * 8 bytes	Per-element null flags

Header Size: 8 + ((num_elements + 63) / 64) * 8 bytes

Array Element Data

Elements are stored contiguously after the header:

• Fixed-width elements: Stored with their natural width (1, 2, 4, or 8 bytes)
• Variable-width elements: Stored as 8-byte offset+size pairs

Element Offset: header_size + element_index * element_size

Data Region Size: Rounded up to nearest 8-byte boundary

Array Element Sizes

Element Type	Element Size
bool	1 byte
int8	1 byte
int16	2 bytes
int32	4 bytes
int64	8 bytes
float32	4 bytes
float64	8 bytes
string/binary	8 bytes (offset+size)
array/map/struct	8 bytes (offset+size)

Map Binary Layout

Maps store key-value pairs as two separate arrays:

+------------------+------------------+------------------+
|  Keys Array Size |   Keys Array     |   Values Array   |
+------------------+------------------+------------------+
|     8 bytes      |   Variable size  |   Variable size  |

Map Structure

Field	Size	Description
Keys Array Size	8 bytes	Total size of keys array in bytes
Keys Array	Variable	Full array structure for keys
Values Array	Variable	Full array structure for values

Keys Array Offset: base_offset + 8 Values Array Offset: base_offset + 8 + keys_array_size

Both keys and values arrays follow the standard array binary layout.

Nested Struct Layout

Nested structs are stored as complete row structures within the variable data region:

1. Parent field slot contains offset+size pointing to nested row
2. Nested row has its own null bitmap and field slots
3. Supports arbitrary nesting depth

Parent Row:
+----------------+------------------+------------------+
|  Null Bitmap   |  ... Slots ...   |  Nested Row Data |
+----------------+------------------+------------------+
                        |                    ^
                        |  offset+size       |
                        +------------------->+

Nested Row:
+----------------+------------------+------------------+
|  Null Bitmap   |  Field Slots     |  Variable Data   |
+----------------+------------------+------------------+

---

Compact Row Format (Java Only)

The compact format provides space-efficient encoding with additional optimizations. It is currently implemented in Java only.

Note: The compact format is still under development and may not be stable yet.

Design Principles

1. Natural Width Storage: Fixed-size fields use their natural byte width instead of 8 bytes
2. Alignment-Based Field Sorting: Fields are sorted by alignment requirements to minimize padding
3. Conditional Null Bitmap: Null bitmap is omitted when all fields are non-nullable
4. Inline Fixed-Size Structs: Nested structs with all fixed-size fields are stored inline

Compact Row Binary Layout

+------------------+------------------+-----+------------------+----------------+------------------+
|  Field 0 Value   |  Field 1 Value   | ... |  Field N-1 Value | Null Bitmap    |  Variable Data   |
+------------------+------------------+-----+------------------+----------------+------------------+
|  W0 bytes        |  W1 bytes        |     |  WN-1 bytes      | B bytes (opt)  |  Variable size   |

Key Differences from Standard Format

1. Field Slot Sizes: Each field uses its natural width (Wi = type width or 8 for variable)
2. Null Bitmap Position: Placed after field slots, can borrow alignment padding
3. Field Order: Fields are sorted by alignment (8-byte → 4-byte → 2-byte → 1-byte → variable)
4. Conditional Bitmap: Skipped entirely if all fields are non-nullable

Null Bitmap (Compact)

• Size: (num_nullable_fields + 7) / 8 bytes (byte-aligned, not 8-byte aligned)
• Skipped: When all fields are primitive/non-nullable
• Position: After all fixed-size field slots, can use alignment padding space

Field Sorting Algorithm

Fields are sorted to minimize padding and optimize alignment:

Priority order (highest to lowest):
1. Fields with 8-byte alignment (int64, float64, variable-width)
2. Fields with 4-byte alignment (int32, float32)
3. Fields with 2-byte alignment (int16)
4. Fields with 1-byte alignment (int8, bool)

Within each alignment group, larger fields come first.

Fixed-Size Struct Inlining

Nested structs with all fixed-size fields are stored inline in the parent row:

Standard Format (nested struct with 2 int32 fields):

Parent slot: [offset (4 bytes) | size (4 bytes)]  → Points to nested row (8+ bytes elsewhere)

Compact Format (same nested struct):

Parent slot: [int32 field 0 | int32 field 1]  → 8 bytes total, inline

This eliminates the offset+size indirection for fixed-size nested structures.

Fixed-Width Calculation

A field's fixed width is determined recursively:

• Primitive types: Natural byte width (1, 2, 4, or 8)
• Struct types: Sum of all child fixed widths (if all children are fixed-width)
• Variable types (string, array, map): Returns -1 (uses 8-byte offset+size slot)

fixed_width(field) =
  if primitive: type_width
  if struct and all_children_fixed: header_bytes + sum(fixed_width(child) for each child)
  else: -1 (variable, uses 8-byte slot)

Compact Array Binary Layout

+------------------+------------------+------------------+
|  Element Count   |   Null Bitmap    |   Element Data   |
+------------------+------------------+------------------+
|     4 bytes      | B bytes (opt)    |   Variable size  |

Compact Array Header

Field	Size	Description
Element Count	4 bytes	Number of elements (int32)
Null Bitmap	(count + 7) / 8 bytes (opt)	Per-element null flags

Header Size Calculation:

header_size = 4 + (element_nullable ? (num_elements + 7) / 8 : 0)

// Round to 8-byte boundary only if element width is 8-byte aligned
if (fixed_width % 8 == 0):
    header_size = round_to_8_bytes(header_size)

Key Differences from Standard Array

1. Element Count: 4 bytes instead of 8 bytes
2. Null Bitmap: Byte-aligned, skipped if elements are non-nullable
3. Fixed-Size Structs: Inline storage for fixed-width struct elements

---

Common Specifications

The following specifications apply to both standard and compact formats.

Type Encoding

Primitive Types

Type	Width	Encoding
bool	1 byte	0x00 (false) or 0x01 (true)
int8	1 byte	Two's complement
int16	2 bytes	Two's complement, little-endian
int32	4 bytes	Two's complement, little-endian
int64	8 bytes	Two's complement, little-endian
float32	4 bytes	IEEE 754 single precision
float64	8 bytes	IEEE 754 double precision

Temporal Types

Type	Width	Encoding
timestamp	8 bytes	Microseconds since Unix epoch (int64)
date32	4 bytes	Days since Unix epoch (int32)
duration	8 bytes	Duration in microseconds (int64)

String and Binary

• Encoding: UTF-8 for strings, raw bytes for binary
• Storage: Offset+size pair in field slot, data in variable region
• Padding: Data padded to 8-byte alignment (standard) or natural alignment (compact)

Null Handling

Row Null Handling

• Null fields have their corresponding bit set to 1 in the null bitmap
• Field slot contents are undefined for null fields (standard) or zeroed (compact)
• Reading a null field returns a null/empty value indicator

Array Null Handling

• Null elements have their corresponding bit set to 1 in the array's null bitmap
• Element data is undefined for null elements
• Compact format: Bitmap skipped if elements are non-nullable

Variable-Width Null Semantics

When reading variable-width data from a null field:

• Returns size of -1 or equivalent null indicator
• No data access is performed

Alignment and Padding

Standard Format Alignment

1. Null Bitmap: Size rounded to 8-byte boundary
2. Field Slots: Always 8 bytes each
3. Variable Data: Each value padded to 8-byte boundary
4. Array Data: Total data region padded to 8-byte boundary

Compact Format Alignment

1. Field Slots: Natural width (1, 2, 4, or 8 bytes)
2. Null Bitmap: Byte-aligned, placed after fields
3. Variable Data: Padded to 8-byte boundary only when needed
4. Header: May use relaxed alignment for smaller overhead

Padding Bytes

• All padding bytes must be set to zero
• Ensures deterministic serialization output
• Prevents information leakage from uninitialized memory

Size Calculations

Standard Row Size

row_size = bitmap_size + num_fields * 8 + variable_data_size

where:
  bitmap_size = ((num_fields + 63) / 64) * 8
  variable_data_size = sum of (padded_size for each variable field)
  padded_size = ((size + 7) / 8) * 8

Compact Row Size

row_size = fixed_region_size + bitmap_size + variable_data_size

where:
  fixed_region_size = sum of (fixed_width(field) or 8 for each field)
  bitmap_size = all_non_nullable ? 0 : (num_nullable_fields + 7) / 8
  // May be rounded to 8-byte boundary if has variable fields

Standard Array Size

array_size = header_size + data_size

where:
  header_size = 8 + ((num_elements + 63) / 64) * 8
  data_size = ((num_elements * element_size + 7) / 8) * 8

Compact Array Size

array_size = header_size + data_size

where:
  header_size = 4 + (element_nullable ? (num_elements + 7) / 8 : 0)
  // header_size rounded to 8 if element_width % 8 == 0
  data_size = num_elements * element_width

Map Size

map_size = 8 + keys_array_size + values_array_size

Summary Tables

Layout Summary

Component	Standard Format	Compact Format
Row Header	((N + 63) / 64) * 8 bytes	0 or (N + 7) / 8 bytes (at end)
Row Field Slots	N * 8 bytes	sum(field_widths) bytes
Array Header	8 + ((E + 63) / 64) * 8 bytes	4 + (E + 7) / 8 bytes (if nullable)
Array Elements	E * element_size (8-aligned)	E * element_width
Map Header	8 bytes	8 bytes
Offset+Size Pair	8 bytes (32-bit offset + size)	8 bytes (same)

Where N = number of fields, E = number of elements

Type Width Summary

Category	Storage Width	Standard Slot	Compact Slot
bool	1 byte	8 bytes	1 byte
int8	1 byte	8 bytes	1 byte
int16	2 bytes	8 bytes	2 bytes
int32	4 bytes	8 bytes	4 bytes
int64	8 bytes	8 bytes	8 bytes
float32	4 bytes	8 bytes	4 bytes
float64	8 bytes	8 bytes	8 bytes
string/binary	Variable	8 bytes	8 bytes
array	Variable	8 bytes	8 bytes
map	Variable	8 bytes	8 bytes
struct	Variable	8 bytes	inline or 8

Implementation Notes

Endianness

• All multi-byte integers are stored in little-endian format
• Floating-point values use native IEEE 754 representation

Memory Safety

• Writers must zero padding bytes to prevent information leakage
• Readers must validate offsets and sizes before accessing data
• Buffer bounds checking is required for untrusted input

Performance Considerations

Standard Format:

• Fixed 8-byte slots enable O(1) field access with simple arithmetic
• 8-byte alignment optimizes CPU cache line usage
• Best for cross-language interoperability

Compact Format:

• Smaller row sizes reduce memory bandwidth
• Field sorting minimizes padding waste
• Inline structs eliminate pointer chasing
• Relaxed alignment may have slight CPU overhead on some architectures

When to Use Each Format

Scenario	Recommended Format
Cross-language data exchange	Standard
Java-only, memory-constrained	Compact
Many small primitive fields	Compact
Many nested fixed-size structs	Compact
Maximum read performance	Standard
Interoperability with C++/Python	Standard