From data to insights: Intelligent entity grouping at scale

Modern software systems contain data objects that naturally form patterns and relationships, but these structures remain hidden to traditional processing techniques as systems grow more complex. Conventional approaches like manual categorization, rule-based systems, and regex patterns fail at scale because they cannot capture semantic meaning or adapt to real-world complexity.

By combining embeddings, which convert entities into vector representations that capture meaning and similarity, with clustering algorithms that automatically discover natural groupings within that semantic space, you can encode semantics to understand what things are about, not just what they look like.

This article presents a production-ready pipeline that uses embeddings and unsupervised clustering to intelligently group entities, including practical implementation details for retries, caching, and error handling.

What are embeddings?

An embedding is a numerical representation of data that preserves semantic meaning in geometric relationships.

Think of it like this: imagine you're mapping people on a piece of paper. People with similar interests sit close together. People with different interests sit far apart. That physical distance encodes meaning.

Embeddings work the same way, but in high-dimensional space instead of 2D paper. An embedding has thousands of dimensions. Each dimension captures a different aspect of meaning. Collectively, they create a rich representation where:

• Similar concepts end up close together
• Dissimilar concepts end up far apart
• The distance between vectors encodes semantic similarity

Modern embeddings come from large language models trained on billions of text examples. These models have "learned" semantic relationships across vast amounts of text.

When you give the models new text, they synthesize what they've learned to create meaningful embeddings. The richer your input data, the better the embedding. The model learns from all the context you provide.

Why Clustering?

Clustering is grouping data without labels. You don't tell the system what groups should exist, it discovers them. Clustering helps to find underlying relationships in your data and also to reduce the complexity of large data sets by reducing the number of dimensions of the data.

K-means clustering is one of the simplest, quickest, and most effective clustering algorithms. And, it works well with embeddings.

Learn more about k-means clustering and other clustering algorithms in this article.

Architecture of the data pipeline

We built a complete data pipeline that handled data preparation, profile construction, embedding generation, clustering, and interpretation.

Here's how intelligent entity grouping works end-to-end:

• Stage 1: Data preparation
• Stage 2: Profile construction
• Stage 3: Embedding generation
• Stage 4: Clustering
• Stage 5: Interpretation
• Stage 6: Caching and serving

Stage 1: Data preparation

The goal of this stage is to fetch all relevant entities with their attributes. This is the foundation of the data pipeline: quality data means quality embeddings means quality clusters.

Step 1: Data Preparation
├─ Fetch entities with their attributes
├─ Filter out inactive or irrelevant entities
└─ Aggregate related data

The following SQL is a generic query template:

-- Generic query template - adapt table/column names to your schema
SELECT
    main_entity.id,
    main_entity.primary_identifier,    -- e.g., name, title, username
    main_entity.primary_category,      -- e.g., type, department, role
    COUNT(DISTINCT related_data.id) as data_richness_score,
    STRING_AGG(DISTINCT category_table.name, ', ') as grouped_categories,
    STRING_AGG(DISTINCT related_data.description, '. ') as aggregated_details
FROM main_entity
LEFT JOIN entity_relation ON main_entity.id = entity_relation.entity_id
LEFT JOIN related_data ON related_data.id = entity_relation.related_id
LEFT JOIN category_table ON related_data.category_id = category_table.id
WHERE main_entity.is_active = true                    -- Filter: active records only
    AND related_data.status IN ('approved', 'verified')  -- Filter: quality data only
GROUP BY main_entity.id, main_entity.primary_identifier, main_entity.primary_category
HAVING COUNT(DISTINCT related_data.id) > 0            -- Require: at least 1 data point
ORDER BY data_richness_score DESC                     -- Prioritize: richer entities first

Consider these design decisions for this SQL code:

• WHERE is_active = true: Skip deleted/inactive entities
• HAVING data_richness_score > 0: Only entities with sufficient data to cluster
• STRING_AGG(): Combines multiple values into rich context for embedding
• Ordered by richness: Prioritize entities with more data

To adapt this SQL code, make these updates:

Stage 2: Profile construction

The goal of this stage is to convert entity attributes into coherent text for embedding.

Step 2: Profile Construction
├─ Combine attributes into coherent text
├─ Maintain semantic richness
└─ Structure: name + categories + achievements

The buildEntityProfile function below transforms structured entity data into embedding-ready text. Place this function in your data processing layer, calling it for each entity before generating embeddings. The function follows a deliberate structure: identity first (name/title), then classifications (categories/tags), followed by rich contextual details, and finally quantitative metrics. This ordering matches how humans conceptualize entities and helps embedding models understand semantics. The config-driven design lets you adapt to any domain—customers, products, articles—without changing the core logic.

// Generic profile builder - customize field mappings for your domain
function buildEntityProfile(entity, config = {}) {
    const parts = [];

    // 1. Primary identifier (required) - the entity's unique name/title
    const idField = config.identifierField || 'name';
    if (entity[idField]) {
        parts.push(`${config.identifierLabel || 'Name'}: ${entity[idField]}`);
    }

    // 2. Classifications (optional) - categories, types, tags
    const categoryField = config.categoryField || 'categories';
    if (entity[categoryField]?.length > 0) {
        const categories = Array.isArray(entity[categoryField])
            ? entity[categoryField].join(', ')
            : entity[categoryField];
        parts.push(`${config.categoryLabel || 'Categories'}: ${categories}`);
    }

    // 3. Rich contextual data (optional) - descriptions, achievements, attributes
    const detailsField = config.detailsField || 'details';
    if (entity[detailsField]) {
        parts.push(`${config.detailsLabel || 'Details'}: ${entity[detailsField]}`);
    }

    // 4. Quantitative signals (optional) - counts, scores, metrics
    const metricField = config.metricField || 'metric_count';
    if (entity[metricField]) {
        parts.push(`${config.metricLabel || 'Activity'}: ${entity[metricField]} items`);
    }

    // 5. Additional fields - extend as needed
    if (config.additionalFields) {
        for (const [field, label] of Object.entries(config.additionalFields)) {
            if (entity[field]) {
                parts.push(`${label}: ${entity[field]}`);
            }
        }
    }

    return parts.join('. ');
}

// Example usage for different domains:

// E-commerce customers:
const customerProfile = buildEntityProfile(customer, {
    identifierField: 'customer_name',
    identifierLabel: 'Customer',
    categoryField: 'purchase_categories',
    categoryLabel: 'Interests',
    detailsField: 'purchase_history',
    detailsLabel: 'Purchase History',
    metricField: 'total_orders',
    metricLabel: 'Orders'
});
// Output: "Customer: John Doe. Interests: Electronics, Books.
//          Purchase History: iPhone 14, MacBook Pro, React Guide. Orders: 12 items."

// Content/Articles:
const articleProfile = buildEntityProfile(article, {
    identifierField: 'title',
    identifierLabel: 'Title',
    categoryField: 'tags',
    categoryLabel: 'Topics',
    detailsField: 'summary',
    detailsLabel: 'Summary',
    metricField: 'view_count',
    metricLabel: 'Views'
});
// Output: "Title: Introduction to Machine Learning. Topics: AI, Tutorial, Python.
//          Summary: A comprehensive guide to ML fundamentals. Views: 15000 items."

Stage 3: Embedding generation

The goal of this stage is to convert text to semantic vectors (embeddings).

Step 3: Embedding Generation
├─ Call embedding API or model
├─ Batch processing for efficiency
├─ Handle failures gracefully
└─ Cache to avoid re-computation

The following code provides a generic embedding generation function along with a provider-specific implementation for IBM watsonx.ai using the Granite slate-125m-english-rtrvr model. Add this to your service layer, calling generateEmbedding (or generateWatsonxEmbedding for watsonx.ai) with the profile text from Stage 2. The watsonx.ai configuration demonstrates how to work with IBM's embedding API, including project ID authentication and the 768-dimensional output optimized for retrieval tasks.

// Generic embedding generation - adapt to your provider (OpenAI, Cohere, Vertex AI, etc.)
async function generateEmbedding(text, config = {}) {
    // Configuration with sensible defaults
    const {
        model = 'text-embedding-3-large',     // Your embedding model
        endpoint = 'https://api.openai.com/v1/embeddings',  // API endpoint
        apiKey = process.env.EMBEDDING_API_KEY,  // API credentials
        timeout = 30000,                       // 30 second timeout
        maxRetries = 3                         // Retry attempts
    } = config;

    const payload = {
        model: model,
        input: [text],
        encoding_format: 'float'               // or 'base64' depending on provider
    };

    const response = await apiClient.post(endpoint, payload, {
        headers: {
            'Authorization': `Bearer ${apiKey}`,
            'Content-Type': 'application/json'
        },
        timeout: timeout
    });

    // Extract embedding from response (adapt to your provider's format)
    const embedding = response.data?.data?.[0]?.embedding;
    if (!Array.isArray(embedding) || embedding.length === 0) {
        throw new Error('Invalid embedding response format');
    }

    return embedding;
}

// Provider-specific adapters:

// IBM Granite on Watsonx.ai Configuration
const watsonxConfig = {
    model: 'ibm/slate-125m-english-rtrvr',  // Granite embedding model
    endpoint: 'https://us-south.ml.cloud.ibm.com/ml/v1/text/embeddings',
    apiKey: process.env.WATSONX_API_KEY,
    projectId: process.env.WATSONX_PROJECT_ID,  // Required for Watsonx
    // Dimensions: 768 (slate-125m-english-rtrvr)
    // Optimized for retrieval and semantic similarity tasks
};

// Watsonx-specific embedding function
async function generateWatsonxEmbedding(text) {
    const payload = {
        model_id: watsonxConfig.model,
        inputs: [text],
        project_id: watsonxConfig.projectId,
        parameters: {
            truncate_input_tokens: 512  // Adjust based on your needs
        }
    };

    const response = await apiClient.post(watsonxConfig.endpoint, payload, {
        headers: {
            'Authorization': `Bearer ${watsonxConfig.apiKey}`,
            'Content-Type': 'application/json',
            'Accept': 'application/json'
        },
        timeout: 30000
    });

    // Watsonx returns embeddings in a different format
    const embedding = response.data?.results?.[0]?.embedding;
    if (!Array.isArray(embedding) || embedding.length === 0) {
        throw new Error('Invalid Watsonx embedding response format');
    }

    return embedding;
}

Learn more about embeddings in this article, Calculating vector embeddings for semantic search and RAG.

Storing embeddings in a vector database

While the in-memory caching shown earlier works for smaller datasets, production systems with thousands or millions of entities benefit from dedicated vector databases. Vector databases like Milvus, Pinecone, Weaviate, or PostgreSQL with pgvector are purpose-built to store high-dimensional vectors and perform efficient similarity searches.

Vector databases work differently than traditional databases. Instead of searching by exact matches, they excel at finding similar vectors quickly—even with millions stored.

To use a vector database in this pipeline, you need to save embeddings to the database right after generating them (Stage 3). Later, when you need to cluster (Stage 4), you load embeddings from the database instead of regenerating them. For updates, only generate new embeddings for changed entities and reuse the stored vectors for everything else.

Production Patterns - Resilience

Production embedding pipelines face real-world challenges: APIs experience temporary outages, rate limits throttle requests, and network connections drop unexpectedly. The following four patterns address these challenges, ensuring your pipeline continues processing even when individual components fail. Implement these patterns in your embedding generation service to build a robust system that degrades gracefully under adverse conditions.

• Pattern 1: Retry with exponential backoff
• Pattern 2: Batch processing with rate limiting
• Pattern 3: Caching to avoid re-computation
• Pattern 4: Fallback for resilience

Pattern 1: Retry with exponential backoff

APIs occasionally restart, experience temporary hiccups, or enforce rate limits and these are expected behaviors, not failures. Exponential backoff handles transient errors gracefully by waiting progressively longer between retry attempts, giving the API time to recover while ensuring a single failure doesn't block your entire pipeline.

async function retryWithBackoff(fn, maxRetries = 3) {
    let lastError;

    for (let attempt = 0; attempt <= maxRetries; attempt++) {
        try {
            return await fn();
        } catch (err) {
            lastError = err;

            // Classify error
            if (!isTransientError(err) || attempt === maxRetries) {
                throw err;  // Permanent error or out of retries
            }

            // Exponential backoff: 100ms, 200ms, 400ms, 800ms
            const delayMs = 100 * Math.pow(2, attempt);
            await sleep(delayMs);
        }
    }

    throw lastError;
}

function isTransientError(err) {
    // Network errors: temporary
    if (['ECONNREFUSED', 'ETIMEDOUT'].includes(err.code)) return true;

    // Server errors (5xx): usually temporary
    if (err.response?.status >= 500) return true;

    // Rate limiting (429): temporary
    if (err.response?.status === 429) return true;

    // Everything else: probably permanent
    return false;
}

Pattern 2: Batch processing with rate limiting

When processing hundreds or thousands of entities, you need to balance speed against API rate limits. Batch processing achieves this by running multiple requests in parallel within each batch (for speed) while adding delays between batches (to respect rate limits). This approach creates predictable, manageable throughput—for example, with batchSize=2 and delayMs=1000, you safely process approximately 30 requests per minute.

async function generateBatchEmbeddings(texts, batchSize = 2, delayMs = 1000) {
    const embeddings = [];
    const errors = [];

    for (let i = 0; i < texts.length; i += batchSize) {
        const batch = texts.slice(i, i + batchSize);

        // Process batch in parallel
        const results = await Promise.all(
            batch.map(text =>
                generateEmbedding(text)
                    .then(embedding => ({ embedding, error: null }))
                    .catch(error => ({ embedding: null, error }))
            )
        );

        // Collect results
        for (const result of results) {
            if (result.error) {
                errors.push(result.error);
            } else {
                embeddings.push(result.embedding);
            }
        }

        // Rate limiting between batches
        if (i + batchSize < texts.length) {
            await sleep(delayMs);
        }
    }

    return { embeddings, errors };
}

Pattern 3: Caching to avoid re-computation

Embedding API calls cost money, consume time, and count against rate limits; yet entity attributes typically change slowly. Caching embeddings eliminates redundant API calls by storing computed vectors and reusing them until the underlying data changes or the cache expires. This pattern dramatically reduces costs and latency for repeat operations.

const embeddingsCache = new Map();
const CACHE_TTL = 24 * 60 * 60 * 1000;  // 24 hours

async function getEmbeddingWithCache(entity) {
    const cacheKey = `entity_${entity.id}`;

    // Check cache first
    if (embeddingsCache.has(cacheKey)) {
        return embeddingsCache.get(cacheKey);
    }

    // Generate if not cached
    try {
        const profileText = buildEntityProfile(entity);
        const embedding = await generateEmbedding(profileText);
        embeddingsCache.set(cacheKey, embedding);
        return embedding;
    } catch (err) {
        // Fallback: generate deterministic embedding
        return generateFallbackEmbedding(entity);
    }
}

Pattern 4: Fallback for resilience

When the embedding API is unavailable—whether due to outages, rate limiting, or network issues, a fallback mechanism keeps your pipeline running. The fallback generates a deterministic embedding from entity attributes (same input always produces the same output). While lower quality than API-generated embeddings, fallback vectors still capture basic categorical similarity, ensuring one failure doesn't block your entire pipeline.

function generateFallbackEmbedding(entity) {
    // Create a deterministic embedding from attributes
    const embedding = new Array(4096).fill(0);

    // Hash categories into embedding space
    if (entity.categories) {
        for (const category of entity.categories) {
            let hash = 0;
            for (let i = 0; i < category.length; i++) {
                hash = ((hash << 5) - hash) + category.charCodeAt(i);
            }
            const idx = Math.abs(hash) % 4096;
            embedding[idx] = Math.min(embedding[idx] + 0.5, 1.0);
        }
    }

    // Weight by importance (attribute count)
    if (entity.attribute_count) {
        const weight = Math.min(entity.attribute_count / 100, 1.0);
        for (let i = 0; i < embedding.length; i++) {
            embedding[i] *= (0.7 + 0.3 * weight);
        }
    }

    // Normalize
    const norm = Math.sqrt(embedding.reduce((s, v) => s + v * v, 0));
    return norm > 0 ? embedding.map(v => v / norm) : embedding;
}

Stage 4: Clustering

The goal of this stage is to group similar entities together.

Step 4: Clustering
├─ Apply K-means algorithm
├─ Group by semantic similarity
└─ Produce cluster assignments

We use K-means clustering, which works in four steps:

• Step 1: Initialize - Pick K random entities as cluster centers.
• Step 2: Assign - Move each entity to its nearest cluster center.
• Step 3: Recalculate - Move each center to the mean of its entities.
• Step 4: Repeat - Until clusters stabilize (convergence).

The ClusteringEngine class below implements K-means clustering from scratch. Instantiate it with your desired number of clusters (K) and call the cluster method with your embedding data. The engine handles initialization, iterative refinement, and convergence detection automatically.

class ClusteringEngine {
    constructor(k = 5, maxIterations = 50) {
        this.k = k;
        this.maxIterations = maxIterations;
    }

    // Distance between two vectors
    euclideanDistance(vec1, vec2) {
        return Math.sqrt(
            vec1.reduce((sum, val, idx) =>
                sum + Math.pow(val - vec2[idx], 2), 0
            )
        );
    }

    // Pick K random data points as centers
    initializeCenters(data) {
        const centers = [];
        const indices = new Set();

        while (centers.length < this.k && centers.length < data.length) {
            const idx = Math.floor(Math.random() * data.length);
            if (!indices.has(idx)) {
                indices.add(idx);
                centers.push([...data[idx].embedding]);
            }
        }

        return centers;
    }

    // Assign each entity to nearest center
    assignToClusters(data, centers) {
        const assignments = new Array(data.length);

        for (let i = 0; i < data.length; i++) {
            let minDistance = Infinity;
            let closestCluster = 0;

            for (let j = 0; j < centers.length; j++) {
                const distance = this.euclideanDistance(
                    data[i].embedding,
                    centers[j]
                );
                if (distance < minDistance) {
                    minDistance = distance;
                    closestCluster = j;
                }
            }

            assignments[i] = closestCluster;
        }

        return assignments;
    }

    // Move centers to mean of assigned points
    recalculateCenters(data, assignments, centers) {
        const newCenters = Array(centers.length)
            .fill(null)
            .map(() => Array(data[0].embedding.length).fill(0));
        const counts = new Array(centers.length).fill(0);

        for (let i = 0; i < data.length; i++) {
            const clusterIdx = assignments[i];
            for (let j = 0; j < data[i].embedding.length; j++) {
                newCenters[clusterIdx][j] += data[i].embedding[j];
            }
            counts[clusterIdx]++;
        }

        // Average
        for (let i = 0; i < newCenters.length; i++) {
            if (counts[i] > 0) {
                for (let j = 0; j < newCenters[i].length; j++) {
                    newCenters[i][j] /= counts[i];
                }
            }
        }

        return newCenters;
    }

    computeInertia(data, assignments, centers) {
        let inertia = 0;

        for (let i = 0; i < data.length; i++) {
            const distance = this.euclideanDistance(
                data[i].embedding,
                centers[assignments[i]]
            );
            inertia += distance * distance;
        }

        return inertia;
    }

    // Main clustering
    cluster(data) {
        if (data.length === 0) return { clusters: [], centers: [] };

        let centers = this.initializeCenters(data);
        let assignments = null;
        let previousInertia = Infinity;
        let iteration = 0;

        for (iteration = 0; iteration < this.maxIterations; iteration++) {
            assignments = this.assignToClusters(data, centers);
            const inertia = this.computeInertia(data, assignments, centers);

            // Check convergence
            if (Math.abs(previousInertia - inertia) < 1e-6) {
                break;  // Converged
            }

            previousInertia = inertia;
            centers = this.recalculateCenters(data, assignments, centers);
        }

        // Build output
        const clusters = Array(this.k).fill(null).map(() => []);
        for (let i = 0; i < data.length; i++) {
            clusters[assignments[i]].push(data[i]);
        }

        return {
            clusters,
            centers,
            iterations: iteration + 1,
            inertia: previousInertia
        };
    }
}

Stage 5: Interpretation

The goal of this stage is to convert raw clusters into human-readable insights.

Step 5: Interpretation
├─ Extract topics from cluster members
├─ Generate human-readable labels
├─ Calculate quality metrics
└─ Identify trending themes

Raw cluster output is meaningless: clusters[0] = [entity1, entity2, entity3, ...]

We need to understand what these entities have in common.

The functions below extract dominant themes from cluster members and generate human-readable labels. Call extractClusterTopics to identify the most frequent categories within a cluster, then pass those topics to generateClusterLabel to create a descriptive summary suitable for display in dashboards or reports.

// Extract dominant themes from cluster members (generic version)
function extractClusterTopics(clusterMembers, config = {}) {
    const {
        categoryField = 'categories',       // Field containing categories/tags
        topN = 5,                           // Number of top themes to extract
        minFrequency = 2                    // Minimum occurrences to be considered
    } = config;

    const categoryFreq = {};

    // Count category/theme occurrences across all cluster members
    for (const member of clusterMembers) {
        const categories = member[categoryField];

        if (categories) {
            // Handle both array and string formats
            const categoryList = Array.isArray(categories)
                ? categories
                : categories.split(',').map(c => c.trim());

            for (const category of categoryList) {
                if (category) {  // Skip empty strings
                    categoryFreq[category] = (categoryFreq[category] || 0) + 1;
                }
            }
        }
    }

    // Get top N by frequency, filtering by minimum threshold
    const topCategories = Object.entries(categoryFreq)
        .filter(([_, count]) => count >= minFrequency)
        .sort((a, b) => b[1] - a[1])
        .slice(0, topN)
        .map(([name, count]) => ({ name, count, percentage: (count / clusterMembers.length) * 100 }));

    return topCategories;
}

// Generate human-readable cluster label (generic version)
function generateClusterLabel(clusterMembers, topics, clusterId, config = {}) {
    const {
        metricField = 'metric_count',       // Field for quantitative metric
        metricLabel = 'items',              // Label for the metric
        includePercentage = false,          // Show topic coverage percentage
        maxTopics = 2                       // Max topics in label
    } = config;

    const memberCount = clusterMembers.length;

    // Calculate average metric if available
    let metricSummary = '';
    if (clusterMembers[0]?.[metricField] !== undefined) {
        const avgMetric = Math.round(
            clusterMembers.reduce((sum, m) => sum + (m[metricField] || 0), 0) / memberCount
        );
        metricSummary = `, ~${avgMetric} ${metricLabel} each`;
    }

    // Build label from dominant topics
    const topicStr = topics
        .slice(0, maxTopics)
        .map(t => includePercentage ? `${t.name} (${Math.round(t.percentage)}%)` : t.name)
        .join(' & ');

    return `${topicStr} (${memberCount} entities${metricSummary})`;
}

// Example usage for different domains:

// E-commerce customers:
const customerTopics = extractClusterTopics(cluster, {
    categoryField: 'purchase_categories',
    topN: 5,
    minFrequency: 3
});
const customerLabel = generateClusterLabel(cluster, customerTopics, 0, {
    metricField: 'total_orders',
    metricLabel: 'orders',
    maxTopics: 2
});
// Output: "Electronics & Fashion (45 entities, ~12 orders each)"

// Content articles:
const articleTopics = extractClusterTopics(cluster, {
    categoryField: 'tags',
    topN: 5,
    minFrequency: 2
});
const articleLabel = generateClusterLabel(cluster, articleTopics, 0, {
    metricField: 'view_count',
    metricLabel: 'views',
    includePercentage: true,
    maxTopics: 2
});
// Output: "Machine Learning (80%) & Python (65%) (23 entities, ~5000 views each)"

Stage 6: Caching and serving

The goal of this stage is to deliver fast responses to users while maintaining data freshness through periodic recomputation.

Step 6: Caching & Serving
├─ Store in-memory cache (fast)
├─ Persist to database (durable)
├─ Track metadata (timing, quality)
└─ Serve to users/applications

Computation is expensive. Caching is cheap. Cache aggressively.

The getGroupings function below implements a two-tier caching strategy. The memory cache provides fast in-process access with a one-hour TTL, while the database cache persists results across application restarts and deployments. This layered approach lets you serve requests in milliseconds while running expensive recomputation on a weekly schedule (or on-demand when data changes significantly). Since clustering results change slowly relative to query frequency, slightly stale data is an acceptable tradeoff for performance.

let cachedResults = null;
let cachedResultsTime = 0;
const MEMORY_CACHE_TTL = 60 * 60 * 1000;  // 1 hour

async function getGroupings(forceRecompute = false) {
    const now = Date.now();

    // Try memory cache
    if (!forceRecompute && cachedResults &&
        now - cachedResultsTime < MEMORY_CACHE_TTL) {
        return {
            ...cachedResults,
            source: 'cache',
            cacheAge: Math.round((now - cachedResultsTime) / 1000)
        };
    }

    // Try database cache
    const dbCached = await loadFromDatabase();
    if (dbCached) {
        cachedResults = dbCached;
        cachedResultsTime = now;
        return { ...dbCached, source: 'database-cache' };
    }

    // Compute fresh
    const results = await computeGroupings();
    cachedResults = results;
    cachedResultsTime = now;

    // Persist
    await saveToDatabase(results);

    return { ...results, source: 'freshly-computed' };
}

Conclusion

We started with a simple question: how do you find meaningful groups in your data without telling the system what those groups should be? The answer lies in combining two powerful techniques: embeddings that capture semantic meaning and clustering that discovers natural structure.

Throughout this article, we constructed a complete entity-grouping pipeline:

1. Data preparation that aggregates rich context from your entities.
2. Profile construction that transforms attributes into embedding-ready text.
3. Embedding generation with production-grade resilience (retries, caching, fallbacks).
4. K-means clustering that groups entities by semantic similarity.
5. Interpretation logic that extracts human-readable insights from raw clusters.
6. Caching and serving that makes results fast and cost-effective.

Each stage is independent and swappable. You can replace the embedding provider, try different clustering algorithms, or adapt the interpretation logic, all without rewriting the entire system.

The real power of this approach isn't in any single component. It's in the combination:

• Embeddings alone give you similarity scores, but no structure
• Clustering alone requires you to define what "similar" means
• Together, embeddings provide the semantic space and clustering finds the natural boundaries within it

The techniques in this article aren't new. Embeddings have been around for over a decade. K-means dates back to the 1950s. What's changed is accessibility: high-quality embeddings are now available through simple API calls, and the infrastructure to run these pipelines at scale is mature and well documented.

This means you can focus on the interesting questions: What entities matter to your business? What attributes capture their essence? What will you do once you understand the natural groupings in your data?

The groupings are already there, latent in your data. Embeddings and clustering just make them visible.

Next steps

If you're ready to implement this in your own system, here's a suggested path:

1. Start small: Pick a single entity type with 100-500 records. Run the pipeline manually. Inspect the clusters. Do they make sense?

2. Iterate on profile construction: The quality of your embeddings depends heavily on what text you feed them. Experiment with different attribute combinations.

3. Tune K empirically: Use the elbow method and silhouette scores to find the right number of clusters for your data. There's no universal answer.

4. Add production hardening gradually: Start with basic retries. Add caching when you hit rate limits. Implement fallbacks when you need reliability guarantees.

5. Monitor and evolve: Track cluster quality over time. Are clusters stable? Are they actionable? Let real world feedback guide refinements.