Observability & Tracing

Every mission, agent instance, and delegate invocation is instrumented with execution logs, OpenTelemetry traces, cost metrics, and a correlated delegation tree. Query them programmatically via REST or MCP to monitor agent behavior, debug failures, and optimize spend.

Overview

Cognethics instruments three observability layers:

Execution logs — agent prompts, tool calls, responses, and state transitions per mission session
OpenTelemetry traces — distributed traces with delegation-tree parent–child correlation (trace_id, span_id)
Cost telemetry — per-agent token counts, USD spend, prompt-cache hits, and attribution by model and user

All data is per-tenant scoped and queryable in real time.

Execution Logs

Fetch mission execution logs

List all events (prompts, tool calls, completions, errors) for a mission session:

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://gtm.cognethics.com/api/agents/missions/observability/?session_id=sess_550e8400"

Response:

{
  "count": 127,
  "results": [
    {
      "id": "evt_550e8400e29b41d4a716446655440000",
      "mission_id": "mis_550e8400e29b41d4a716446655440000",
      "session_id": "sess_550e8400e29b41d4a716446655440000",
      "event_type": "agent_prompt_sent",
      "timestamp": "2026-05-21T15:30:00Z",
      "agent_id": "agt_550e8400e29b41d4a716446655440000",
      "data": {
        "prompt": "Analyze this invoice for line-item discrepancies...",
        "model": "claude-opus-4-7",
        "temperature": 1.0,
        "max_tokens": 4096
      }
    },
    {
      "id": "evt_550e8400e29b41d4a716446655440001",
      "mission_id": "mis_550e8400e29b41d4a716446655440000",
      "session_id": "sess_550e8400e29b41d4a716446655440000",
      "event_type": "tool_call",
      "timestamp": "2026-05-21T15:30:05Z",
      "agent_id": "agt_550e8400e29b41d4a716446655440000",
      "data": {
        "tool_name": "document_search",
        "tool_input": { "query": "invoice line items May 2026" },
        "tool_use_id": "toolu_01ARZ3NDEKTSV4RRFFQ69G40F"
      }
    },
    {
      "id": "evt_550e8400e29b41d4a716446655440002",
      "mission_id": "mis_550e8400e29b41d4a716446655440000",
      "session_id": "sess_550e8400e29b41d4a716446655440000",
      "event_type": "tool_result",
      "timestamp": "2026-05-21T15:30:06Z",
      "agent_id": "agt_550e8400e29b41d4a716446655440000",
      "data": {
        "tool_name": "document_search",
        "content": "Found 3 invoices matching date range...",
        "error": null
      }
    }
  ]
}

Event types:

agent_prompt_sent — Agent received a prompt and submitted a response
tool_call — Agent invoked an MCP tool
tool_result — Tool returned a result (or error)
agent_completed — Agent retired successfully
agent_failed — Agent terminated with an error
delegation_started — Parent delegated work to a child agent
delegation_completed — Child completed task and folded results back
escalation_requested — Child escalated decision above its authority

Query by event type

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://gtm.cognethics.com/api/agents/missions/observability/?session_id=sess_550e8400&event_type=tool_call"

Query a time range

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://gtm.cognethics.com/api/agents/missions/observability/?session_id=sess_550e8400&date_from=2026-05-21T00:00:00Z&date_to=2026-05-21T23:59:59Z"

Distributed Tracing

OpenTelemetry trace federation

Every mission is instrumented with OpenTelemetry spans. Each span carries:

trace_id — correlated across the entire mission and all delegated child agents
span_id — unique within the trace
parent_span_id — links to the caller's span (parent agent or orchestrator)
Standard OTEL attributes — duration, status, events, attributes

Traces let you:

Correlate logs across parent and child agents (same trace_id)
Visualize the execution tree (parent–child span_id nesting)
Pinpoint latency hotspots (span duration)
Detect failures in real time (span status = ERROR)

Delegation-tree visualization

Fetch a visual tree of all delegation relationships in a mission:

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://gtm.cognethics.com/api/agents/traces/delegation-tree/?mission_id=mis_550e8400&format=json"

Response:

{
  "mission_id": "mis_550e8400e29b41d4a716446655440000",
  "root_agent": {
    "id": "agt_550e8400e29b41d4a716446655440000",
    "name": "Chief Auditor",
    "persona": "auditor",
    "status": "completed",
    "span_id": "span_550e8400e29b41d4a716446655440000",
    "trace_id": "trace_550e8400e29b41d4a716446655440000",
    "duration_ms": 2345,
    "children": [
      {
        "id": "agt_550e8400e29b41d4a716446655440001",
        "name": "Invoice Analyzer",
        "persona": "analyst",
        "status": "completed",
        "parent_span_id": "span_550e8400e29b41d4a716446655440000",
        "span_id": "span_550e8400e29b41d4a716446655440001",
        "trace_id": "trace_550e8400e29b41d4a716446655440000",
        "duration_ms": 1200,
        "children": []
      },
      {
        "id": "agt_550e8400e29b41d4a716446655440002",
        "name": "Risk Assessor",
        "persona": "risk_officer",
        "status": "completed",
        "parent_span_id": "span_550e8400e29b41d4a716446655440000",
        "span_id": "span_550e8400e29b41d4a716446655440002",
        "trace_id": "trace_550e8400e29b41d4a716446655440000",
        "duration_ms": 890,
        "children": []
      }
    ]
  }
}

Use this to:

Trace execution flow — see which agent completed which tasks
Debug failures — identify the first agent to error
Correlate with logs — match span_id to tool calls in the execution log

Cost Telemetry

Per-agent token counts

Fetch token usage (input, cache creation, cache read, output) for each agent:

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://gtm.cognethics.com/api/agents/costs/token-distribution/?mission_id=mis_550e8400"

Response:

{
  "mission_id": "mis_550e8400e29b41d4a716446655440000",
  "total_input_tokens": 45000,
  "total_cache_creation_tokens": 8000,
  "total_cache_read_tokens": 12000,
  "total_output_tokens": 8500,
  "agents": [
    {
      "agent_id": "agt_550e8400e29b41d4a716446655440000",
      "agent_name": "Chief Auditor",
      "model": "claude-opus-4-7",
      "input_tokens": 25000,
      "cache_creation_tokens": 8000,
      "cache_read_tokens": 2000,
      "output_tokens": 3200,
      "usd_cost": 15.20,
      "cache_hit_rate": 0.18
    },
    {
      "agent_id": "agt_550e8400e29b41d4a716446655440001",
      "agent_name": "Invoice Analyzer",
      "model": "claude-opus-4-7",
      "input_tokens": 12000,
      "cache_creation_tokens": 0,
      "cache_read_tokens": 10000,
      "output_tokens": 3100,
      "usd_cost": 7.50,
      "cache_hit_rate": 0.45
    }
  ]
}

Cache visibility. The token distribution includes:

cache_creation_tokens — tokens in prompts that established a cache page
cache_read_tokens — tokens reused from previously cached pages
cache_hit_rate — fraction of input tokens served from cache

Higher cache hit rates reduce cost and latency.

Cost timeline

Fetch cumulative spend over time:

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://gtm.cognethics.com/api/agents/costs/timeline/?mission_id=mis_550e8400&granularity=minute"

Response:

{
  "mission_id": "mis_550e8400e29b41d4a716446655440000",
  "granularity": "minute",
  "timeline": [
    {
      "timestamp": "2026-05-21T15:30:00Z",
      "cumulative_usd": 0.05,
      "cumulative_input_tokens": 500,
      "cumulative_output_tokens": 100
    },
    {
      "timestamp": "2026-05-21T15:31:00Z",
      "cumulative_usd": 2.30,
      "cumulative_input_tokens": 8200,
      "cumulative_output_tokens": 1800
    },
    {
      "timestamp": "2026-05-21T15:32:00Z",
      "cumulative_usd": 5.10,
      "cumulative_input_tokens": 18000,
      "cumulative_output_tokens": 3200
    }
  ]
}

Use this to spot cost spikes and correlate them with execution events (tool calls that consume many tokens).

Memory usage analytics

Fetch per-agent persistent-memory usage (bytes, entries, last modified):

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://gtm.cognethics.com/api/agents/pj/analytics/memory/?mission_id=mis_550e8400"

Response:

{
  "agents": [
    {
      "agent_id": "agt_550e8400e29b41d4a716446655440000",
      "agent_name": "Chief Auditor",
      "memory_type": "primary",
      "entry_count": 23,
      "total_bytes": 18420,
      "last_written": "2026-05-21T15:31:45Z",
      "entries": [
        {
          "key": "invoice_classification_rules",
          "value_bytes": 1250,
          "created_at": "2026-05-20T09:00:00Z",
          "last_modified": "2026-05-21T15:28:00Z"
        },
        {
          "key": "vendor_risk_matrix",
          "value_bytes": 3100,
          "created_at": "2026-05-19T14:30:00Z",
          "last_modified": "2026-05-21T15:30:15Z"
        }
      ]
    }
  ]
}

Governance Analytics

For governance-driven missions, fetch policy-evaluation metrics:

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://gtm.cognethics.com/api/agents/pj/analytics/governance/?mission_id=mis_550e8400"

Returns:

Policies evaluated (count)
Policies violated (count and names)
Decisions blocked vs. allowed
Escalations triggered and reasons

Use Cases

Debug a failed agent

Fetch the execution log for the mission
Find the first error event (event_type = agent_failed or tool_result with error)
Use the trace_id to find related parent and child spans
Correlate with tool calls to identify which MCP operation failed

Optimize spending

Fetch the cost timeline for a mission
Identify the minute(s) with the steepest spend curve
Fetch the execution log for those timestamps
Check cache_hit_rate in token distribution — if low, prompt caching needs tuning
Look at tool calls to spot redundant or batchable operations

Audit a delegation chain

Fetch the delegation-tree report
Verify each child completed successfully (status = completed)
Cross-check span_id against the execution log to confirm fold-back results

API Reference

List mission execution logs

GET /api/agents/missions/observability/

Query parameters:

session_id (required) — mission session ID
event_type (optional) — filter by event type
agent_id (optional) — filter by agent
date_from, date_to (optional) — ISO 8601 timestamp range
page (optional) — page number (default 1)
page_size (optional) — results per page (default 50)

Response: 200 OK with paginated execution log entries.

Get detailed mission observability

GET /api/agents/missions/observability/<session_id>/

Response: 200 OK with summary metrics and linked resources (traces, costs).

Fetch delegation-tree report

GET /api/agents/traces/delegation-tree/?mission_id=<id>&format=json

Query parameters:

mission_id (required) — mission to visualize
format (optional) — json (default) or mermaid (for diagramming)

Response: 200 OK with the tree structure.

Fetch cost timeline

GET /api/agents/costs/timeline/?mission_id=<id>&granularity=minute

Query parameters:

mission_id or instance_id (required)
granularity (optional) — second, minute, hour, day (default: minute)
date_from, date_to (optional) — ISO 8601 range

Response: 200 OK with timeline datapoints.

Fetch token distribution

GET /api/agents/costs/token-distribution/?mission_id=<id>

Response: 200 OK with per-agent token counts and cache metrics.

Fetch memory analytics

GET /api/agents/pj/analytics/memory/?mission_id=<id>

Response: 200 OK with per-agent memory usage.

Fetch governance analytics

GET /api/agents/pj/analytics/governance/?mission_id=<id>

Response: 200 OK with policy-evaluation metrics.

Next Steps

Monitor a mission in real time — poll the execution log and cost timeline
Build a dashboard — fetch observability data every 5 seconds and render the delegation tree
Optimize spend — use token distribution and timeline data to identify and eliminate redundant tool calls
Audit governance decisions — cross-check governance analytics with the audit log

Questions? See the handler catalog or reach our support team through the contact form.