Working with Resources

Work with Resource datasets (mutable state tracking) using OPAL temporal joins. Resources track state changes over time with Valid From/To timestamps. Resources are rarely useful by themselves - their power comes from enriching Events/Intervals with contextual state information through temporal joins.

Use when you need to:

• Enrich spans/logs with service metadata (type, environment, namespace)
• Track database dependencies between services
• Add resource state context to point-in-time events
• Navigate from events to related resources across datasets

Covers temporal join mechanics, three working join verbs (lookup, join, follow), and automatic field matching.

Key Concepts

What Are Resources?

Resources represent mutable state that changes over time:

• Valid From / Valid To timestamps (nanoseconds)
• Track how resource properties evolve (e.g., service environment changes)
• Created from Events/Intervals using make_resource verb
• Different from Intervals (which are immutable completed activities)

Example: OpenTelemetry/Service Resource tracks service metadata:

service_name: "checkoutservice"
service_type: "Service"
environment: "eu01"
Valid From: 2025-01-10T00:00:00Z
Valid To: 2025-01-11T00:00:00Z

Temporal Joins

Critical: Resources use automatic temporal joins based on timestamp overlap:

Span:     |-----------|  (start_time to end_time)
Resource:    |-------------|  (Valid From to Valid To)
Match: YES (overlap exists)

OPAL automatically:

1. Matches timestamps (span time must overlap Resource validity period)
2. Matches field names (e.g., service_name = service_name)
3. No explicit ON clause needed

Performance note: join is faster than lookup for Resource joins (avoids extra left outer join pass).

Join Verbs for Resources

Three join verbs work reliably with Resources:

Pattern 1: lookup - Enrich All Rows

Use when: You want to keep ALL primary dataset rows, with optional Resource enrichment.

# Enrich ALL checkoutservice spans with service metadata
filter service_name = "checkoutservice"
| lookup @service_resource
| make_col svc:service_name,
          svc_type:service_type,
          svc_env:environment,
          dur_ms:duration / 1ms
| topk 10

Behavior:

• Returns all spans (even if no Resource match found)
• Resource fields populated when match exists
• Slower due to two-pass join (band-join + left outer join)

Common MCP parameters:

{
  "primary_dataset_id": "42160967",
  "secondary_dataset_ids": ["42160979"],
  "dataset_aliases": {"service_resource": "42160979"}
}

Pattern 2: join - Filter to Matches Only

Use when: You only want rows WITH Resource matches (faster performance).

# Show only spans where service metadata exists
filter service_name = "checkoutservice"
| join @service_resource
| make_col svc:service_name,
          svc_type:service_type,
          dur_ms:duration / 1ms
| limit 10

Behavior:

• Returns only spans with matching Resources
• Faster than lookup (single-pass band-join)
• Filters out spans with no Resource match

When to use:

• Performance matters
• You don't need rows without Resource context
• Filtering to matches is acceptable

Pattern 3: follow - Navigate to Related Dataset

Use when: You want to find related dataset rows (cross-dataset navigation).

Critical difference: follow returns rows from the FOLLOWED dataset, not the primary dataset!

# Given error logs, show ALL related spans by namespace
filter body ~ /error/i
| follow @spans.service_namespace = namespace
| make_col svc:service_name,
          span:span_name,
          dur_ms:duration / 1000000
| limit 20

Syntax:

follow @dataset.field = primary_field

Behavior:

• Returns rows from @spans (not from logs)
• Matches based on field equality + temporal overlap
• Use case: "Given these logs, show me all related spans"

Important: This is NOT enrichment - you're switching to a different dataset entirely.

Common MCP parameters for follow:

{
  "primary_dataset_id": "42161740",
  "secondary_dataset_ids": ["42160967"],
  "dataset_aliases": {"spans": "42160967"}
}

Common Patterns

Pattern: Service Inventory from Resource

# Count services by type in eu01 environment
filter environment = "eu01"
| make_col svc:service_name,
          svc_type:service_type,
          svc_ns:service_namespace
| statsby count:count(), group_by(svc_type)
| sort desc(count)

Result: 14 Services + 1 Database in eu01

Note: This queries the Resource dataset directly (no join needed).

Pattern: Database Dependencies

# Find which services call which databases
make_col caller:parent_service_name,
         database:service_name,
         env:environment
| statsby count:count(), group_by(caller, database, env)
| sort desc(count)
| topk 10

Dataset: ServiceExplorer/Database Call (Resource)

Result: Shows cartservice → redis, observe-community-mcp → postgresql

Pattern: Enrich Spans with Service Context

# Add service type and environment to high-latency spans
filter duration > 100ms
| join @service_resource
| make_col svc:service_name,
          type:service_type,
          env:environment,
          latency_ms:duration / 1ms
| statsby avg_latency:avg(latency_ms),
          count:count(),
          group_by(svc, type, env)
| sort desc(avg_latency)

Why join here: Filtering to spans with service metadata is acceptable, and it's faster than lookup.

Pattern: Navigate from Logs to Spans

# Find all spans from namespaces that logged errors
filter body ~ /error/i
| follow @spans.service_namespace = namespace
| make_col svc:service_name,
          span:span_name,
          dur_ms:duration / 1000000
| statsby error_span_count:count(),
          avg_latency:avg(dur_ms),
          group_by(svc, span)
| sort desc(error_span_count)

Why follow: We want to see ALL spans from namespaces with errors, not just enrich the error logs.

Dataset: Kubernetes Logs (42161740) following to OpenTelemetry/Span (42160967)

Troubleshooting

Issue: "Field not found" after join

Cause: Resource field name doesn't match primary dataset field.

Solution: Check exact field names in Resource dataset:

discover_context(dataset_id="42160979")

Copy field names exactly (case-sensitive!).

Issue: No Resource matches found

Diagnosis:

1. Check temporal overlap - Resource must be valid during span/event time
2. Verify field name matching (e.g., both have service_name)
3. Confirm Resource data exists for that time range

Solution: Query Resource dataset directly to see available data:

filter service_name = "checkoutservice"
| make_col valid_from:int64("Valid From"),
          valid_to:int64("Valid To")
| limit 5

Issue: "implicit lookup does not support additional arguments"

Error: Tried using ON clause with lookup:

| lookup @resource on service_name = @resource.service_name  ❌

Fix: Remove ON clause - temporal joins are automatic:

| lookup @resource  ✅

Issue: follow returns unexpected dataset

Cause: follow returns rows from the FOLLOWED dataset, not primary.

This is correct behavior:

• lookup/join: Keep primary dataset structure + add Resource fields
• follow: Switch to followed dataset, return matching rows

Solution: Use lookup or join if you want to enrich primary dataset.

Available Resource Datasets

Common Resources in Observe (from discovery):

OpenTelemetry/Service (ID: 42160979):

• service_name (str)
• service_type (str) - "Database", "Service"
• environment (str)
• service_namespace (str)

ServiceExplorer/Database Call (ID: 42160978):

• service_name (str) - Database name
• parent_service_name (str) - Calling service
• environment (str)

Discovery: Use MCP to find Resources:

discover_context("resource", result_type="dataset", interface_filter="resource")

Performance Considerations

Faster: join over lookup

• Documentation recommends join for Resource/Interval joins
• Avoids extra left outer join pass
• Only use lookup when you MUST keep all rows

Temporal join cost:

• OPAL performs interval overlapping band-join
• Matches timestamps + field names automatically
• More expensive than simple equality joins

Best practice: Filter primary dataset first to reduce join volume:

filter service_name = "checkoutservice"  # Filter FIRST
| join @service_resource                 # Then join

Key Takeaways

1. Resources track mutable state over time (Valid From/To timestamps)
2. Power comes from temporal joins, not standalone queries
3. Three working join verbs:
   • lookup - enrich all rows (slower)
   • join - filter to matches (faster)
   • follow - navigate to related dataset
4. Automatic temporal matching - no ON clause needed
5. Performance: Prefer join over lookup when filtering acceptable

References

• Temporal joins use interval overlapping band-join
• set_valid_from and set_valid_to define Resource validity
• join is faster than lookup (single-pass vs two-pass)
• follow requires field-based join condition: follow @dataset.field = primary_field