Python

Programming with Recap.

1. Overview
2. Clients
3. Converters
4. Types
   1. Converting a CST to an AST
   2. Converting an AST to a CST
   3. Working with Schemas

Overview

Recap is written in Python and is designed to be used as a Python library.

There are three main components:

• recap.clients - Recap clients for connecting to systems and reading schemas.
• recap.converters - Recap converters for converting schemas to and from Recap schemas.
• recap.types - Recap’s type system.

If you’re unfamiliar with clients, converters, and types, read Recap’s concepts page.

Clients

Clients connect to remote systems, read schemas, and conver them to Recap schemas.

You can connect using create_client:

from recap.clients import create_client

with create_client('postgresql://user:pass@host:port/dbname') as client:
    dbs = client.ls()
    struct = client.get_schema("testdb", "public", "users")

See the integrations page for each system’s URL format.

Converters

Converters are used to convert to and from Recap schemas. Unlike clients, converters do not connect to remote systems. They just take an in-memory schema and return a Recap schema (or vice-versa).

Here’s how to convert a Protobuf schema to a Recap schema:

from recap.converters.protobuf import ProtobufConverter

proto_schema = """
syntax = "proto3";

message User {
    string name = 1;
    int32 age = 2;
}
"""
recap_schema = ProtobufConverter().to_recap(proto_schema)

Each converter has a to_recap and/or from_recap method. The method parameters vary depending on the converter. See the integrations page for specifics on each converter.

Types

Recap’s Type Spec is implemented in types.py. This code is Recap’s abstract syntax tree (AST). If you want to manipulate schemas, you need to get familar with this file.

The two most important methods are to_dict and from_dict. These methods convert Recap’s concrete syntax tree (CST) to and from Recap’s AST.

We’re getting a bit academic here, so let’s be specific about AST vs. CST. Recap’s AST is defined in its *Type classes (StructType, IntType, and so on). Recap’s CST is contained in a dictionary following the syntax defined in the Type Spec ({"type": "int", "bits": 32}). Type systems have both because the AST is easier to work with, but the CST is easier to serialize.

Converting a CST to an AST

Use from_dict to convert a concrete syntax tree (in a dictionary) to an abstract syntax tree:

from recap.types import from_dict

cst = {
    "type": "struct",
    "fields": [
        {
            "type": "int",
            "bits": 32,
        },
        {
            "type": "string",
            "bytes": 50,
        },
    ],
}
ast = from_dict(cst)

Converting an AST to a CST

Use to_dict to convert an abstract syntax tree to a concrete syntax tree (in a dictionary):

from recap.types import to_dict

ast = StructType(
    fields=[
        IntType(bits=32),
        StringType(bytes_=50),
    ]
)
cst = to_dict(ast)

Working with Schemas

Transforming schemas to add, remove, or modify fields is a common operation. You can manipulate schemas in Recap’s AST, CST, or in the system’s native schema format. Recap doesn’t care.

Let’s look at some AST examples.

There are a lot of examples in Recap’s test_types.py file. Take a look if you want to see more.

Create a Struct

This code creates a struct with an int32 field. The field is required (it’s not nullable and has no default).

from recap.types import StructType, IntType

struct = StructType(
    fields=[
        IntType(bits=32),
    ]
)

Make Fields Optional

This code creates a struct with a single optional field.

from recap.types import StructType, IntType, UnionType, NullType

StructType(
    fields=[
        UnionType(
            default=None,
            types=[
                NullType(),
                IntType(bits=32),
            ],
        )
    ]
)

Optional fields are represented as a union of null and the field type with a default set to None.

Name Fields

Struct fields usually have a name, so let’s add one.

from recap.types import StructType, IntType, UnionType, NullType

StructType(
    fields=[
        UnionType(
            default=None,
            types=[
                NullType(),
                IntType(bits=32),
            ],
            name="age",
        )
    ]
)

Adding a Field

Let’s add a email field to our struct.

from recap.types import StructType, IntType, UnionType, NullType, StringType

ast = StructType(
    fields=[
        UnionType(
            default=None,
            types=[
                NullType(),
                IntType(bits=32),
            ],
            name="age",
        ),
    ]
)

ast.fields.append(StringType(bytes_=50, name="email"))

The email field is required. It’s not nullable and has no default.

Removing a Field

Let’s add a email field to our struct.

from recap.types import StructType, IntType, UnionType, NullType, StringType

ast = StructType(
    fields=[
        UnionType(
            default=None,
            types=[
                NullType(),
                IntType(bits=32),
            ],
            name="age",
        ),
        StringType(bytes_=50, name="email"),
    ]
)

ast.fields = [field for field in ast.fields if field.extra_attrs["name"] != "email"]

This example modifies the AST in-place. The *Types are mutable, but it’s safer to treat the AST objects as immutable by creating new objects.

Modifying a Field

Let’s modify the email length to 100 bytes.

from recap.types import StructType, IntType, UnionType, NullType, StringType

ast = StructType(
    fields=[
        UnionType(
            default=None,
            types=[
                NullType(),
                IntType(bits=32),
            ],
            name="age",
        ),
        StringType(bytes_=50, name="email"),
    ]
)

for field in ast.fields:
    if field.extra_attrs["name"] == "email":
        field.bytes_ = 100