Discover insights with Topics

• Discover what users are trying to accomplish
• Find where users struggle, get confused, or encounter errors
• Track user satisfaction and frustration across interactions
• Pinpoint failures and recurring issues in your AI’s responses

​

How it works

1. Preprocessing — Each trace is formatted into readable text. Messages, tool calls, and nested spans become a narrative.
2. Facets — An LLM analyzes the preprocessed trace and extracts a concise summary based on user intent (Task), emotional tone (Sentiment), and agent behavior (Issues).
3. Topics — Once at least 100 facet summaries are collected, a clustering algorithm groups similar ones. For example, “User wants a refund,” “Requesting a chargeback,” and “Asking for money back” might all become the topic “Refund requests.”
4. Classification — Each trace is labeled with the closest topic (e.g., “Refund requests”). Classifications appear in your logs table, where you can filter, query with SQL, and build evaluation datasets.

• Initially: Existing logs are optionally backfilled with facet summaries.
• Continuously: New logs are processed contiuously as they arrive.
• Daily: Topics are regenerated daily from collected facet summaries. Topic generation requires at least 100 summaries.

​

Enable topics

1. Go to Logs and click Enable topics. A dialog opens showing the three built-in facets that will be activated:
   • Task: Extracts the user’s intent or goal from the conversation (e.g., “Creating a dataset,” “Debugging an API error”).
   • Sentiment: Extracts the user’s emotional tone (e.g., “POSITIVE,” “FRUSTRATED,” “NEUTRAL”).
   • Issues: Identifies problems with agent behavior or responses (e.g., “Tool call failed,” “Incomplete answer”).
2. Choose whether to Apply to existing traces or only to new traces.
3. Click Enable topics.

​

Review insights

​

Examine specific traces

​

View topic distributions

​

Cluster a filtered subset

​

Track trends over time

​

Act on findings

​

Build datasets from topics

1. Go to Logs and click Filter.
2. Select Classifications and choose the classification you want to filter by. Alternately, click SQL and enter a filter clause. See the SQL reference for more query patterns.
   Filter by specific topic

   Filter by multiple topics

   Combine topics across facets

   classifications.Task.label = "Dataset creation"
   
3. Select the logs you want to include.
4. Click + Dataset and choose an existing dataset or create a new one.

• “Error Investigation” tasks → test your error handling
• Negative sentiment interactions → improve responses
• “Pricing Questions” → evaluate your pricing explanations

​

Score logs based on topics

import braintrust from "braintrust";
import { z } from "zod";

const project = braintrust.projects.create({ name: "my-project" });

project.scorers.create({
  name: "Checkout experience",
  slug: "checkout-experience",
  description: "Flag traces with negative checkout experiences",
  parameters: z.object({
    trace: z.any(),
  }),
  handler: async ({ trace }) => {
    if (!trace) return { score: null };

    const spans = await trace.getSpans();
    const rootSpan = spans.find((s) => s.span_id === s.root_span_id);
    if (!rootSpan) return { score: null };

    const classifications = rootSpan.classifications || {};
    const taskClassification = (classifications.Task || [{}])[0];
    const sentimentClassification = (classifications.Sentiment || [{}])[0];

    if (
      taskClassification.label === "Checkout Flow" &&
      sentimentClassification.label === "NEGATIVE"
    ) {
      return {
        score: 0,
        metadata: { reason: "Negative sentiment during checkout" },
      };
    }

    return { score: 1 };
  },
});

1. Save the code to a file and push it:
   TypeScript

   Python

   bt functions push topic_scorer.ts
   
2. Go to Settings > Automations and click + Rule.
3. Select your scorer, set Scope to Trace, configure the sampling rate, and click Create rule.

​

Assign topics for review

1. Go to Logs and click Filter.
2. Select Classifications and choose the classification you want to filter by. Alternately, click SQL and enter a filter clause. See the SQL reference for more query patterns.
   Filter by specific topic

   Filter by multiple topics

   Combine topics across facets

   classifications.Task.label = "Dataset creation"
   
3. Select the logs you want to assign.
4. Select Assign and choose a team member.

​

Configure and manage topics

​

Adjust topics automation

• Filter: Restricts which traces are processed. Use the basic filter UI or enter a SQL expression directly.
• Sampling rate: Percentage of incoming traces that get processed (default: 100%).
• Idle time: Seconds after the last trace before the automation is run again (default: 600).

​

Check automation status

​

Create custom facets

• Domain-specific categories: Your logs have patterns that built-in topic maps don’t capture.
• Too many uncategorized traces: The built-in topic maps aren’t extracting relevant summaries.
• Wrong level of detail: You need more specific categorization (e.g., distinguish between different API endpoints instead of just “API request”).
• Business-specific needs: Track patterns unique to your product (e.g., “Feature requests,” “Pricing questions,” “Integration issues”).

1. Go to Logs and select Topics > + Create facet.
2. Give your topic map a name and description. Example:
   • “Churn risk”
   • “Topic map for assessing customer churn risk based on conversations”
3. Choose a preprocessor to transform your trace data.
   • Select Preprocessor > Thread (default) to format traces as conversation threads.
   • Or select + Custom preprocessor to write a JavaScript function that filters or transforms your data. Common patterns:
     Extract user messages

     Format with metadata

     Filter to errors only

     // Extract only user messages from a conversation
     function userMessagesOnly(span) {
       const messages = span.input?.messages || [];
       const userMessages = messages
         .filter(m => m.role === 'user')
         .map(m => m.content)
         .join('\n\n');
     
       return userMessages || '(No user messages found)';
     }
     
4. Enter a prompt with clear instructions for what to extract. Example:
   Based on this conversation, assess the churn risk for this customer.
   
   Consider:
   
   - Frustration level and language used (complaints, strong negative words)
   - Whether their issue was resolved satisfactorily
   - Mentions of competitors, alternatives, or cancellation
   - Overall satisfaction signals (thanks, happy, vs angry, disappointed)
   - Severity and recurrence of issues
   
   Classify as:
   - LOW RISK: Satisfied customer, issue resolved, positive interaction
   - MEDIUM RISK: Some frustration but issue handled, no major red flags
   - HIGH RISK: Frustrated customer, unresolved issues, or mentions of dissatisfaction
   - CRITICAL: Explicitly mentioned canceling, switching to competitor, or very angry
   
   Respond with the label followed by a colon and the key risk indicators (one sentence).
   
   Examples:
   - "LOW_RISK: User thanked the agent and confirmed their billing question was answered."
   - "HIGH_RISK: User expressed frustration about repeated API errors and said this is unacceptable."
   - "CRITICAL: User stated they are considering switching to a competitor if issues persist."
   
5. If necessary, enter a case-insensitive regex to exclude facet outputs from being included in topic generation. For example, if the facet output is 'NONE', a regex of '^NONE' will exclude it from being used in topic generation.
6. Click Test to verify extraction quality on sample traces.
7. Choose whether to Apply to existing traces or only to new traces.
   This option triggers processing across all automated facets, not just this one. Logs already processed for a given facet are skipped, but facets you previously opted out of backfilling may get processed.
8. Click Create.

​

Pause a facet

1. Go to Logs.
2. Select Topics and choose the facet.
3. Click Pause.

​

Troubleshooting

​

Next steps

• Query classifications with SQL
• Monitor trends with dashboards and charts
• Create evaluation datasets from specific topics
• Set up online scoring based on classifications