Overview

The Annotation Format Schema (AFS) is an open format created and used by Agentation for capturing UI feedback in a way that AI coding agents can reliably parse and act on. Think of it like smart Figma comments for your running app — persistent annotations attached to specific elements, with threads, status tracking, resolution workflows, and structured metadata that agents can actually understand.

This spec defines the annotation object shape. Tools can emit annotations in this format, and agents can consume them regardless of how they were created.

What This Unlocks

A structured schema isn't just about clean data — it enables entirely new workflows:

  • Two-way communication — Agents can reply to annotations, asking “Should this be 24px or 16px?” and get responses in the same thread
  • Status tracking — See what's pending, acknowledged, resolved, or dismissed at a glance
  • Cross-page queries — “What annotations do I have?” works across your entire site
  • Bulk operations — “Clear all annotations” or “Show me blocking issues only”
  • Persistent history — Feedback survives page refreshes and browser sessions

Without a schema, feedback is fire-and-forget. With one, it becomes a conversation.

Design Goals

  • Agent-readable — Structured data that LLMs can parse without guessing
  • Framework-agnostic — Works with any UI, though React gets extra context
  • Tool-agnostic — Any tool can emit, any agent can consume
  • Human-authored — Designed for feedback from humans (or automated reviewers)
  • Minimal core — Few required fields, many optional for richer context

Annotation Object

An annotation represents a single piece of feedback attached to a UI element.

Note: The server may add metadata fields (sessionId, createdAt, updatedAt) when syncing annotations.

Required Fields

{
  id: string;           // Unique identifier (e.g. "ann_abc123")
  comment: string;      // Human feedback ("Button is misaligned")
  elementPath: string;  // CSS selector path ("body > main > button.cta")
  timestamp: number;    // Unix timestamp (ms)
  x: number;            // % of viewport width (0-100)
  y: number;            // px from document top (or viewport if isFixed)
  element: string;      // Tag name ("button", "div", "input")
}

Recommended Fields

{
  url: string;          // Page URL where annotation was created
  boundingBox: {        // Element position at annotation time
    x: number;
    y: number;
    width: number;
    height: number;
  };
}

Optional Context Fields

{
  // React-specific (when available)
  reactComponents: string;  // Component tree ("App > Dashboard > Button")

// Element details
  cssClasses: string;       // Class list ("btn btn-primary disabled")
  computedStyles: string;   // Key CSS properties
  accessibility: string;    // ARIA attributes, role
  nearbyText: string;       // Visible text in/around element
  selectedText: string;     // Text highlighted by user

// Feedback classification
  intent: "fix" | "change" | "question" | "approve";
  severity: "blocking" | "important" | "suggestion";

// Annotation kind (defaults to "feedback")
  kind: "feedback" | "placement" | "rearrange";

// Layout mode: placement data
  placement: {
    componentType: string;  // e.g. "Hero", "Card", "Navigation"
    width: number;          // px
    height: number;         // px
    scrollY: number;        // scroll position when placed
    text?: string;          // optional label text
  };

// Layout mode: rearrange data
  rearrange: {
    selector: string;       // CSS selector of the section
    label: string;          // human-readable label
    tagName: string;        // HTML tag name
    originalRect: { x: number; y: number; width: number; height: number };
    currentRect: { x: number; y: number; width: number; height: number };
  };
}

Lifecycle Fields

{
  status: "pending" | "acknowledged" | "resolved" | "dismissed";
  resolvedAt: string;       // ISO timestamp
  resolvedBy: "human" | "agent";
  thread: ThreadMessage[];  // Back-and-forth conversation
}

Browser Component Fields

These optional fields are set by the Agentation browser component for UI rendering:

{
  isFixed: boolean;         // Element has fixed/sticky positioning
  isMultiSelect: boolean;   // Created via drag selection
  fullPath: string;         // Full DOM path (vs shorter elementPath)
  nearbyElements: string;   // Info about nearby DOM elements
}

Full TypeScript Definition

type Annotation = {
  // Required
  id: string;
  comment: string;
  elementPath: string;
  timestamp: number;
  x: number;            // % of viewport width (0-100)
  y: number;            // px from document top (or viewport if isFixed)
  element: string;      // Tag name ("button", "div")

// Recommended
  url?: string;
  boundingBox?: {
    x: number;
    y: number;
    width: number;
    height: number;
  };

// Optional context
  reactComponents?: string;
  cssClasses?: string;
  computedStyles?: string;
  accessibility?: string;
  nearbyText?: string;
  selectedText?: string;

// Browser component fields
  isFixed?: boolean;       // Element has fixed/sticky positioning
  isMultiSelect?: boolean; // Created via drag selection
  fullPath?: string;       // Full DOM path
  nearbyElements?: string; // Info about nearby elements

// Feedback classification
  intent?: "fix" | "change" | "question" | "approve";
  severity?: "blocking" | "important" | "suggestion";

// Annotation kind
  kind?: "feedback" | "placement" | "rearrange";

// Layout mode data
  placement?: {
    componentType: string;
    width: number;
    height: number;
    scrollY: number;
    text?: string;
  };
  rearrange?: {
    selector: string;
    label: string;
    tagName: string;
    originalRect: { x: number; y: number; width: number; height: number };
    currentRect: { x: number; y: number; width: number; height: number };
  };

// Lifecycle
  status?: "pending" | "acknowledged" | "resolved" | "dismissed";
  resolvedAt?: string;
  resolvedBy?: "human" | "agent";
  thread?: ThreadMessage[];
};

type ThreadMessage = {
  id: string;
  role: "human" | "agent";
  content: string;
  timestamp: number;
};

Example Annotation

{
  "id": "ann_k8x2m",
  "comment": "Button is cut off on mobile viewport",
  "elementPath": "body > main > .hero-section > button.cta",
  "timestamp": 1705694400000,
  "x": 45.5,
  "y": 480,
  "element": "button",
  "url": "http://localhost:3000/landing",
  "boundingBox": { "x": 120, "y": 480, "width": 200, "height": 48 },
  "reactComponents": "App > LandingPage > HeroSection > CTAButton",
  "cssClasses": "cta btn-primary",
  "nearbyText": "Get Started Free",
  "intent": "fix",
  "severity": "blocking",
  "status": "pending"
}

Layout Mode Example

Layout mode annotations use the kind field to distinguish from regular feedback:

{
  "id": "ann_d3s1gn",
  "comment": "Place a Hero component here",
  "elementPath": "body",
  "timestamp": 1709510400000,
  "x": 50,
  "y": 200,
  "element": "body",
  "kind": "placement",
  "placement": {
    "componentType": "Hero",
    "width": 800,
    "height": 400,
    "scrollY": 0,
    "text": "Hero"
  },
  "status": "pending"
}

Implementations

Tools that emit or consume this format:
Agentation (React) Click-to-annotate toolbar for React apps
Agentation MCP Server Exposes annotations to Claude Code and other MCP clients

Building an Implementation

To emit Agentation Format annotations from your tool:

  1. Capture the required fields: id, comment, elementPath, timestamp, x, y, element
  2. Add recommended fields for better agent accuracy: url, boundingBox
  3. For React apps, traverse the fiber tree to get reactComponents
  4. Output as JSON for MCP/API consumption, or markdown for chat pasting

See the Agentation source for reference implementations of element detection and React component traversal.

Why This Format?

Existing agent protocols (MCP, A2A, ACP) standardize tools and messaging, but they don't define a UI feedback grammar. They rely on whatever structured context you feed them.

This format fills that gap: a portable wire format specifically for "human points at UI, agent needs to find and fix the code." We hope it's useful to others building similar tools.