Agent skill

langsmith-tracing

LangSmith tracing and debugging setup for LLM applications. Configure observability, capture traces, and enable debugging for LangChain/LangGraph agents.

Stars 514
Forks 31

Install this agent skill to your Project

npx add-skill https://github.com/a5c-ai/babysitter/tree/main/library/specializations/ai-agents-conversational/skills/langsmith-tracing

SKILL.md

langsmith-tracing

Configure LangSmith observability and tracing for LLM applications built with LangChain and LangGraph frameworks.

Overview

LangSmith is the managed observability suite by LangChain that provides:

  • Dashboards and alerting for LLM applications
  • Human-in-the-loop evaluation capabilities
  • Deep LangChain/LangGraph integration
  • Run Tree model for nested traces
  • MCP connectivity to Claude, VSCode

Capabilities

Core Tracing Setup

  • Initialize LangSmith client and API configuration
  • Configure project/workspace settings
  • Set up trace collection and sampling
  • Enable debug logging for agent execution

Integration Patterns

  • LangChain chain tracing with automatic instrumentation
  • LangGraph workflow state tracking
  • Custom span creation for non-LangChain code
  • Parent-child trace relationships

Debugging Features

  • Fetch execution traces for analysis
  • Query run history and metadata
  • Export traces for offline analysis
  • Compare runs across different versions

Usage

Environment Setup

bash
# Set required environment variables
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=<your-api-key>
export LANGCHAIN_PROJECT=<project-name>

Python Integration

python
from langsmith import Client, traceable
from langchain.callbacks.tracers import LangChainTracer

# Initialize client
client = Client()

# Use @traceable decorator for custom functions
@traceable(name="custom_operation")
def my_function(input_data):
    # Your logic here
    return result

# Initialize tracer for LangChain
tracer = LangChainTracer(project_name="my-project")

# Use with LangChain chains
chain.invoke(input, config={"callbacks": [tracer]})

Trace Retrieval

python
# Fetch traces from LangSmith
runs = client.list_runs(
    project_name="my-project",
    start_time=datetime.now() - timedelta(hours=24),
    execution_order=1,  # Root runs only
    error=False,  # Successful runs only
)

for run in runs:
    print(f"Run ID: {run.id}")
    print(f"Latency: {run.latency_p99}")
    print(f"Tokens: {run.total_tokens}")

Task Definition

When used in a babysitter process, this skill produces:

javascript
const langsmithTracingTask = defineTask({
  name: 'langsmith-tracing-setup',
  description: 'Configure LangSmith tracing for the application',

  inputs: {
    projectName: { type: 'string', required: true },
    apiKeyEnvVar: { type: 'string', default: 'LANGCHAIN_API_KEY' },
    samplingRate: { type: 'number', default: 1.0 },
    enableDebug: { type: 'boolean', default: false }
  },

  outputs: {
    configured: { type: 'boolean' },
    projectUrl: { type: 'string' },
    artifacts: { type: 'array' }
  },

  async run(inputs, taskCtx) {
    return {
      kind: 'skill',
      title: `Configure LangSmith tracing for ${inputs.projectName}`,
      skill: {
        name: 'langsmith-tracing',
        context: {
          projectName: inputs.projectName,
          apiKeyEnvVar: inputs.apiKeyEnvVar,
          samplingRate: inputs.samplingRate,
          enableDebug: inputs.enableDebug,
          instructions: [
            'Verify LangSmith API credentials are available',
            'Create or validate project configuration',
            'Set up tracing instrumentation in codebase',
            'Configure sampling rate and debug settings',
            'Verify traces are being captured correctly'
          ]
        }
      },
      io: {
        inputJsonPath: `tasks/${taskCtx.effectId}/input.json`,
        outputJsonPath: `tasks/${taskCtx.effectId}/result.json`
      }
    };
  }
});

Applicable Processes

  • llm-observability-monitoring
  • agent-evaluation-framework
  • react-agent-implementation
  • conversation-quality-testing
  • regression-testing-agent

External Dependencies

  • LangSmith account and API key
  • LangChain Python library
  • langsmith Python package

References

Related Skills

  • SK-OBS-002 langfuse-integration
  • SK-OBS-003 phoenix-arize-setup
  • SK-OBS-004 opentelemetry-llm

Related Agents

  • AG-OPS-004 observability-engineer
  • AG-SAF-004 agent-evaluator

Expand your agent's capabilities with these related and highly-rated skills.

a5c-ai/babysitter

gsd-tools

Central utility skill for GSD operations. Provides config parsing, slug generation, timestamps, path operations, and orchestrates calls to other specialized skills. Acts as the unified entry point that the original gsd-tools.cjs provided via its lib/ modules (commands, config, core, init).

514 31
Explore
a5c-ai/babysitter

model-profile-resolution

Resolve model profile (quality/balanced/budget) at orchestration start and map agents to specific models. Enables cost/quality tradeoffs by selecting appropriate AI models for each agent role.

514 31
Explore
a5c-ai/babysitter

verification-suite

Plan structure validation, phase completeness checks, reference integrity verification, and artifact existence confirmation. Provides the structured verification layer ensuring GSD artifacts are well-formed and complete.

514 31
Explore
a5c-ai/babysitter

state-management

STATE.md reading, writing, and field-level updates. Provides cross-session state persistence via .planning/STATE.md with structured fields for current task, completed phases, blockers, decisions, and quick tasks.

514 31
Explore
a5c-ai/babysitter

git-integration

Git commit patterns, formats, and conventions for GSD methodology. Provides atomic commits per task, structured commit messages, planning file commits, branch management, and milestone tag operations.

514 31
Explore
a5c-ai/babysitter

frontmatter-parsing

YAML frontmatter parsing and manipulation for .planning/ documents. Provides read, write, update, query, and validation operations on frontmatter blocks in GSD markdown artifacts.

514 31
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results