Skip to content
Back to top
View this page

# Binary Data (Blobs)

Blobs provide efficient binary data storage with typed layouts and zero-copy NumPy integration.

When to use: Use blobs for binary data like images, meshes, and raw buffers. `BlobArray` for typed arrays, `BlobPack` for structured multi-region data.

## Blob vs BlobId

Viper offers two DSM types for binary data:

Type | Use Case | Storage
--- | --- | ---
`blob` | Small data (thumbnails, icons) | Inline in document
`blob_id` | Large data (textures, meshes) | Database blob API

### blob (Inline)

A `blob` field stores binary data directly in the document. No special API needed.

### blob_id (Reference)

A `blob_id` is a SHA-1 hash referencing a blob managed by the Database blob API. This requires the dedicated blob API:

    >>> layout = BlobLayout()
    >>> content = ValueBlob(bytes([1, 2, 3, 4, 5]))

    >>> blob_id = db.create_blob(layout, content)
    >>> blob_id
    f07d73c81ed1a91165a75c5cc22253cc7895a8b2

    >>> db.blob(blob_id)
    blob(5)

    >>> blob_id in db.blob_ids()
    True

    >>> info = db.blob_info(blob_id)
    >>> info.size()
    5

Content-addressable: The `blob_id` is computed from layout + content (SHA-1). Identical content always produces the same `blob_id`.

Constraint: A document referencing a `blob_id` cannot be committed unless the blob exists in the database. Always call `create_blob()` before using the `blob_id` in a document.

## ValueBlob

A `ValueBlob` holds inline binary data. To pull the bytes back out, use the `bytes()` builtin (the buffer protocol) — there is no `.bytes()` method:

    >>> data = bytes([1, 2, 3, 4, 5])
    >>> blob = ValueBlob(data)

    >>> bytes(blob)
    b'\x01\x02\x03\x04\x05'

    >>> len(blob)
    5

## ValueBlobId

A `ValueBlobId` references external binary data. The id is computed from the layout and content (deterministic SHA-1):

    >>> layout = BlobLayout()
    >>> content = ValueBlob(bytes([1, 2, 3, 4]))
    >>> blob_id = ValueBlobId(layout, content)
    >>> blob_id
    6b8f3ca756046be29244d9bdb6b5ca5c00468ad5

    >>> ValueBlobId.try_parse("6b8f3ca756046be29244d9bdb6b5ca5c00468ad5")
    6b8f3ca756046be29244d9bdb6b5ca5c00468ad5

## BlobLayout - Metadata Everywhere

A `BlobLayout` describes how to interpret blob bytes. This is the Metadata Everywhere principle applied to binary data: the layout is metadata that gives meaning to raw bytes.

    >>> BlobLayout()
    'uchar-1'

    >>> BlobLayout('float', 3)
    'float-3'

    >>> BlobLayout('uint', 3)
    'uint-3'

    >>> BlobLayout('float', 2)
    'float-2'

The layout enables:

- Type-safe interpretation of binary data

- Cross-platform compatibility (endianness handled)

- Validation at decode time

## BlobView (Read-Only)

A `BlobView` interprets an existing blob with a given layout (read-only). When the layout has more than one component per element, indexing returns a tuple:

    >>> import struct
    >>> _buf = bytearray()
    >>> for i in range(100):
    ...     _ = _buf.extend(struct.pack('<fff', float(i), float(i*2), float(i*3)))
    >>> raw = ValueBlob(bytes(_buf))

    >>> view = BlobView(BlobLayout('float', 3), raw)
    >>> view.count()
    100
    >>> view[0]
    (0.0, 0.0, 0.0)
    >>> view[99]
    (99.0, 198.0, 297.0)

Use `BlobView` when you need to read blob data without copying.

## BlobArray (Read-Write)

A `BlobArray` is a typed array backed by a blob. Writes and reads use the NumPy buffer protocol — the array exposes a flat `(N*components,)` view, so reshape it to `(N, components)` to assign per-element tuples:

    >>> import numpy as np
    >>> layout = BlobLayout('float', 3)
    >>> array = BlobArray(layout, 100)

    >>> np_view = np.array(array, copy=False).reshape(100, 3)
    >>> np_view[0] = [1.0, 2.0, 3.0]
    >>> np_view[1] = [4.0, 5.0, 6.0]

    >>> view = BlobView(layout, array.blob())
    >>> view[0]
    (1.0, 2.0, 3.0)
    >>> view[1]
    (4.0, 5.0, 6.0)

## BlobPack - Structured Binary Data

A `BlobPack` groups multiple named regions with different layouts into a single blob. This is ideal for complex structures like 3D meshes.

### Example: 3D Mesh Storage

A mesh has positions, normals, UVs, and triangle indices — each with a different layout. Define the structure with a descriptor, then create the pack:

    >>> descriptor = BlobPackDescriptor()
    >>> descriptor.add_region('positions', BlobLayout('float', 3), 4)
    >>> descriptor.add_region('normals', BlobLayout('float', 3), 4)
    >>> descriptor.add_region('uvs', BlobLayout('float', 2), 4)
    >>> descriptor.add_region('indices', BlobLayout('uint', 3), 2)

    >>> mesh = BlobPack(descriptor)
    >>> len(mesh)
    4

### Fill the Mesh Data

Each region exposes a flat NumPy view; reshape to write per-vertex tuples:

    >>> import numpy as np
    >>> pos = np.array(mesh['positions'], copy=False).reshape(4, 3)
    >>> pos[:] = [[-1.0, -1.0, 0.0], [1.0, -1.0, 0.0],
    ...           [ 1.0,  1.0, 0.0], [-1.0,  1.0, 0.0]]

    >>> normals = np.array(mesh['normals'], copy=False).reshape(4, 3)
    >>> normals[:] = [[0.0, 0.0, 1.0]] * 4

    >>> uvs = np.array(mesh['uvs'], copy=False).reshape(4, 2)
    >>> uvs[:] = [[0.0, 0.0], [1.0, 0.0], [1.0, 1.0], [0.0, 1.0]]

    >>> indices = np.array(mesh['indices'], copy=False).reshape(2, 3)
    >>> indices[:] = [[0, 1, 2], [0, 2, 3]]

### Serialize and Restore

    >>> blob = mesh.blob()
    >>> restored = BlobPack.from_blob(blob)

    >>> np.array(restored['positions'], copy=False).reshape(4, 3)[0].tolist()
    [-1.0, -1.0, 0.0]
    >>> np.array(restored['indices'], copy=False).reshape(2, 3)[1].tolist()
    [0, 2, 3]

### Region Access

    >>> 'positions' in mesh
    True
    >>> 'colors' in mesh
    False

    >>> mesh['positions'].name()
    'positions'
    >>> mesh['positions'].count()
    4
    >>> mesh['positions'].blob_layout()
    'float-3'
    >>> mesh['positions'].data_count()
    12
    >>> mesh['positions'].byte_count()
    48

### Why BlobPack?

Benefit | Description
--- | ---
Single blob | All mesh data in one blob_id
Typed regions | Each region has its own layout
Self-describing | Layout metadata embedded in header
Efficient | Direct memory mapping, no parsing

## NumPy Integration

`BlobArray` implements the Python Buffer Protocol, enabling zero-copy interoperability with NumPy and other array libraries.

### Zero-Copy View

The buffer is exposed as a flat 1D array of components — reshape to give it the geometric shape you want:

    >>> import numpy as np
    >>> layout = BlobLayout('float', 3)
    >>> positions = BlobArray(layout, 100)

    >>> np_view = np.array(positions, copy=False)
    >>> np_view.shape
    (300,)
    >>> np_view.dtype
    dtype('float32')

    >>> np_view.reshape(100, 3).shape
    (100, 3)

### Bidirectional Modifications

Changes through NumPy affect the original BlobArray:

    >>> reshaped = np.array(positions, copy=False).reshape(100, 3)
    >>> reshaped[0] = [10.0, 20.0, 30.0]

    >>> view = BlobView(layout, positions.blob())
    >>> view[0]
    (10.0, 20.0, 30.0)

### Direct Memory Access

    >>> mv = memoryview(positions)
    >>> mv.nbytes
    1200

### BlobPack Regions

`BlobPackRegion` also supports the Buffer Protocol — same flat-then-reshape idiom:

    >>> positions_np = np.array(mesh['positions'], copy=False).reshape(4, 3)
    >>> normals_np = np.array(mesh['normals'], copy=False).reshape(4, 3)
    >>> uvs_np = np.array(mesh['uvs'], copy=False).reshape(4, 2)

    >>> positions_np.shape
    (4, 3)
    >>> uvs_np.shape
    (4, 2)

    >>> positions_np *= 2.0
    >>> positions_np[0].tolist()
    [-2.0, -2.0, 0.0]

### Why Zero-Copy Matters

Scenario | Without Zero-Copy | With Zero-Copy
--- | --- | ---
1M vertices | Copy 12MB | Share pointer
GPU upload | Python → copy → C++ | Direct access
Scientific compute | Data duplication | In-place ops

## Blob in Attachments

Blobs can be used in attachments:

    // Small data inline
    struct Thumbnail {
    uint16 width;
    uint16 height;
    blob data;
    // Inline binary
    };

    // Large data by reference
    struct Texture {
    uint16 width;
    uint16 height;
    blob_id pixels;
    // External reference
    };

## Storing Blobs in Database

### Inline Blobs

Inline blobs are stored directly in the document. The Tuto fixture exposes a `Thumbnail` struct with a `blob` field, attached to `User` as `avatar`:

    >>> user_key = TUTO_A_USER_AVATAR.create_key()

    >>> thumb = TUTO_A_USER_AVATAR.create_document()
    >>> thumb.width = 64
    >>> thumb.height = 64
    >>> thumb.data = ValueBlob(bytes([1, 2, 3, 4, 5]))
    >>> thumb
    {width=64, height=64, data=blob(5)}

    >>> ms = CommitMutableState(db.initial_state())
    >>> ms.attachment_mutating().set(TUTO_A_USER_AVATAR, user_key, thumb)
    >>> avatar_commit = db.commit_mutations("Add avatar", ms)

    >>> db.state(avatar_commit).attachment_getting().get(TUTO_A_USER_AVATAR, user_key)
    Optional({width=64, height=64, data=blob(5)})

### Referenced Blobs (blob_id)

Use the database blob API to store and retrieve. The Tuto fixture exposes a `Texture` struct with a `blob_id` field, attached to `User` as `portrait`:

    >>> mesh_layout = BlobLayout()
    >>> mesh_content = ValueBlob(bytes([10, 20, 30, 40]))
    >>> texture_blob_id = db.create_blob(mesh_layout, mesh_content)

    >>> texture = TUTO_A_USER_PORTRAIT.create_document()
    >>> texture.width = 1024
    >>> texture.height = 1024
    >>> texture.pixels = texture_blob_id

    >>> ms = CommitMutableState(db.state(avatar_commit))
    >>> ms.attachment_mutating().set(TUTO_A_USER_PORTRAIT, user_key, texture)
    >>> portrait_commit = db.commit_mutations("Add portrait", ms)

    >>> db.state(portrait_commit).attachment_getting().get(TUTO_A_USER_PORTRAIT, user_key)
    Optional({width=1024, height=1024, pixels=...})

### Retrieving Blobs

    >>> stored_id = db.create_blob(BlobLayout(), ValueBlob(bytes([10, 20, 30, 40, 50])))
    >>> bytes(db.blob(stored_id))
    b'\n\x14\x1e(2'

    >>> db.read_blob(stored_id, size=2, offset=0)
    blob(2)

### BlobStream (Large Blobs)

For very large blobs, use streaming to avoid loading everything in memory.

Required for blobs > 2GB: The standard `create_blob()` API has a 2GB size limit. Use BlobStream for larger data:

    >>> stream = db.blob_stream_create(BlobLayout('uchar', 1), size=10)

    >>> db.blob_stream_append(stream, ValueBlob(bytes([1, 2, 3, 4, 5])))
    >>> db.blob_stream_append(stream, ValueBlob(bytes([6, 7, 8, 9, 10])))

    >>> stream_blob_id = db.blob_stream_close(stream)
    >>> bytes(db.blob(stream_blob_id))
    b'\x01\x02\x03\x04\x05\x06\x07\x08\t\n'

This is essential for:

- 3D meshes with millions of vertices

- Video/audio data

## When to Use Each Type

Scenario | Recommendation
--- | ---
Thumbnails (< 64KB) | Use `blob` (inline)
Textures (> 1MB) | Use `blob_id` (Database blob API)
Mesh geometry | Use `blob_id` (Database blob API)
Icons, small images | Use `blob` (inline)
Audio/video | Use `blob_id` (Database blob API)

## What’s Next

- Serialization - JSON and binary encoding