Meta Ray-Ban x Poke

How it works

Glasses ──capture──► Mobile App ──POST──► Poke Engine ──inferObject──► Vision LLM
  (DAT SDK)         (Android/iOS)     (ingest trigger)     (analysis)
                      │                                       │
                      ◄──────────── response ──────────────────┘

1. 01Glasses capture a frame via the DAT SDK's captureFrame().
2. 02Mobile app base64-encodes the JPEG and POSTs to Poke's ingest endpoint.
3. 03Poke runs the meta_rayban_bridge.ts script, passing the image to a vision model.
4. 04The LLM returns a scene description; Poke relays it back to the mobile app.

Getting started

Mobile SDK integration

Example implementations for both platforms using the Meta Wearables DAT SDK. Full source files in mobile/.

Android (Kotlin)

val wearables = Wearables(context, WearablesConfig(
    appId = "YOUR_APP_ID",
    appSecret = "YOUR_APP_SECRET"
))

wearables.requestPermissions(
    listOf(CAMERA, STREAMING),
    onGranted = { /* permissions OK */ },
    onDenied = { /* handle denial */ }
)

wearables.connect(SessionConfig(autoReconnect = true)) { session ->
    val frame: ByteArray = session.captureFrame()

    // Downscale and compress
    val scaled = BitmapFactory.decodeByteArray(frame, 0, frame.size)
        .let { Bitmap.createScaledBitmap(it, 1080, ..., true) }
    val jpeg = ByteArrayOutputStream().apply {
        scaled.compress(JPEG, 80, this)
    }.toByteArray()

    // POST to Poke
    postToPoke(jpeg, prompt = "What am I looking at right now?")

iOS (Swift)

let wearables = Wearables(config: WearablesConfig(
    appId: "YOUR_APP_ID",
    appSecret: "YOUR_APP_SECRET"
))

try await wearables.requestPermissions([.camera, .streaming])

let session = try await wearables.connect(
    config: SessionConfig(autoReconnect: true)
)

let frame: Data = try await session.captureFrame()

let scaled = UIImage(data: frame)?
    .jpegData(compressionQuality: 0.8)

let response = try await postToPoke(
    imageData: scaled,
    prompt: "What am I looking at right now?"
)

CameraCaptureAndroid.kt·CameraCaptureIOS.swift

Poke automation script

Runs on every ingest trigger. Decodes the image and asks a vision LLM to describe the scene. Deploy to your Poke project at poke/meta_rayban_bridge.ts.

export const ingestHandler = createServerFn({ method: "POST" })
  .validator((d: unknown) => ingestSchema.parse(d))
  .handler(async ({ data }) => {
    const { image, prompt } = data
    const imageBuffer = Buffer.from(image, "base64")

    const result = await Agent.inferObject({
      model: "claude-3-5-sonnet-v2@2025-04-15",
      system: "Describe what they are looking at in 1-3 concise sentences.",
      prompt: prompt,
      attachments: [{ type: "image", data: imageBuffer, mediaType: "image/jpeg" }],
    })

    return { status: "ok", analysis: result.text, latency_ms: Date.now() - startTime }
  })

meta_rayban_bridge.ts·Trigger config guide

API & handshake protocol

The mobile app sends a POST to the Poke ingest endpoint. Full spec in docs/architecture.md.

Request payload

{
  "image": "<base64-encoded JPEG>",
  "prompt": "What am I looking at right now?",
  "device_id": "abc123-def456",
  "timestamp": "2026-07-13T10:30:00Z",
  "meta": { "app_version": "1.0.0", "sdk_version": "0.5.0" }
}

Response

{
  "status": "ok",
  "analysis": "You are looking at a wooden desk with a laptop, a coffee mug, and a succulent plant.",
  "latency_ms": 1234
}

Error codes

Authentication

Authorization: Bearer pk_xxxxxxxxxxxxxxxx

Generate your API key in the Poke dashboard under Settings > API Keys. Store it in the platform keystore on mobile devices.

Repository