Large language models (LLMs) have become indispensable across various fields, from content creation to automated customer support. As the ecosystem expands with a growing number of open-source models, such as the IBM Granite or Llama family of models, selecting the right model for a specific task has become increasingly complex. Each model excels in different areas, which makes it crucial to compare the LLM outputs systematically.

To facilitate this process, this tutorial provides a reusable codebase to compare multiple open-source LLMs effectively. This code leverages technologies such as LangChain and watsonx.ai to enable seamless integration and querying of various models. Comparing outputs from multiple LLMs streamlines model selection and helps you find the most suitable model for your workloads.

In this tutorial, you explore the importance of comparing outputs from multiple LLMs, examine the challenges involved, and review solutions for a systematic evaluation. You also walk through the code necessary to compare various LLMs to evaluate cost, latency, and quality.

Why compare LLM outputs?

To understand the benefits of comparing LLM outputs, consider the key reasons why comparison is important, along with the challenges of evaluating multiple LLMs.

Consider these 4 core reasons for evaluating LLMs for optimal use:

Task-specific performance. Not all LLMs are created equal. One model may excel at creative writing, another at structured summarization, and yet another at factual question answering. Evaluating outputs across tasks like summarization, question answering (QA), or creative generation ensures that you select the best model for the job.
Bias and variability. Each LLM reflects biases based on its training data. For instance, some LLMs may demonstrate biases in areas like gender stereotypes. These biases can manifest differently across models, impacting decisions in areas like content moderation or research. Identifying and mitigating these biases is crucial for ensuring fairness and accuracy in the outputs.
Cost efficiency. Cost is another important factor. Some LLMs offer high-quality outputs at lower costs, making them more viable for continuous use. Comparing models helps balance the trade-offs between cost and quality, ensuring that resource-intensive tasks don’t unnecessarily strain budgets.
Generalization ability. LLMs trained on specific datasets may struggle to generalize to new, unseen scenarios. By comparing outputs across diverse tasks, you can assess which models adapt best to different situations, ensuring robustness in real-world applications.

What are the challenges when comparing multiple LLMs?

Evaluating multiple LLMs presents several technical challenges:

Consistency in output. LLMs often produce varied and sometimes inconsistent responses to the same query. For example, when summarizing a story, one model might focus on emotional aspects while another emphasizes factual content. Quantifying these differences can be challenging, as quality and relevance are often subjective.
Latency and cost. Running multiple LLMs simultaneously increases compute time and costs, particularly with API-driven models. Latency issues can arise, especially when querying large models or running chains of queries for complex tasks. High API costs can also make large-scale testing expensive.
Evaluation complexity. Choosing the right metrics for comparison is another challenge. While fluency and style might matter more for creative tasks, accuracy and reliability are key for factual QA. Some comparisons require human evaluation, while others can rely on automated metrics like BLEU, ROUGE, or F1, depending on the task.
Scalability. As the number of models and tasks grows, scaling infrastructure to support parallel evaluations becomes complex. Efficiently managing multiple model queries and data pipelines for testing requires significant resources and well-designed systems.
Bias and ethical considerations. Different LLMs are trained on diverse datasets, leading to different biases. Identifying these biases is critical, but it’s also essential to understand how the training data influences fairness, especially in sensitive applications.

Building a codebase for comparing LLMs

To address these challenges, you can build a reusable codebase that allows for efficient comparison of multiple open-source LLMs. This codebase enables you to input queries, run them across various models, and automatically compare the outputs based on selected criteria, such as task-specific performance, cost, and generalization ability.

By using an automated system, you can compare multiple models in parallel, drastically reducing manual effort. This approach ensures faster testing and makes it easier to find the best-performing model for your specific needs. A systematic comparison generates valuable insights into each model's strengths and weaknesses. For example, one model may excel at conversational fluency while another offers more accurate factual responses.

With a modular codebase, you can easily add new models or adjust testing parameters. This flexibility allows you to evaluate newly released models or explore how different prompts affect output quality.

Finally, by comparing outputs, you can optimize workflows by selecting the most suitable LLM for each task.

Architecture and design

The testing framework evaluates different LLM models through the IBM watsonx.ai platform with the following structure:

Core functionality
- Connect to multiple AI models (such as Llama, Mistral, and Granite) through watsonx.ai
- Send test prompts and collect responses
- Compare performance metrics (speed, cost, and quality)
- Generate clear comparison reports
Design considerations
- Use parallel processing for efficiency
- Handle model response formats automatically using the SDK chat API
- Simplify adding new models
- Track token counts and estimated costs
Key components
- Model connection layer: Handle watsonx authentication and configure model parameters
- Testing infrastructure: Run tests in parallel with ThreadPoolExecutor, track response time, calculate costs, and measure response quality
- Analysis and reporting: Compute per-prompt and aggregate statistics across models
Test plan
- Simple prompt: General question answering ("What is the capital of France?")
- Complex prompt: Systems engineering requirements analysis against INCOSE guidelines
- Models tested: ibm/granite-3-8b-instruct, meta-llama/llama-3-8b-instruct, and mistralai/mistral-large

Setting up the development environment

Before building the LLM testing framework, import the required libraries and configure authentication credentials. The following code block sets up imports and environment variables:

# Importing necessary libraries
import os
import re
import time
from langchain_ibm import WatsonxLLM
import concurrent.futures
from typing import List, Dict
from dataclasses import dataclass
from datetime import datetime
import numpy as np

os.environ['WATSONX_APIKEY'] = "ENTER YOUR WATSONX API KEY"
os.environ['WATSONX_PROJECT_ID'] = "ENTER YOUR WATSONX PROJECT ID"

Core response generation and metrics tracking

This section defines 2 critical components: the response generation function and a data structure for tracking metrics.

def generate_response(prompt, model):
    '''
    Generates a response from a given prompt using a specified model.
    '''
    parameters = {
        "decoding_method": "sample",
        "min_new_tokens": 1,
        "max_new_tokens": 4096,
        "stop_sequences": [],
        "repetition_penalty": 1,
        "temperature": 0
    }

    watsonx_llm = WatsonxLLM(
        model_id=model,
        url="https://us-south.ml.cloud.ibm.com",
        project_id=os.environ.get("WATSONX_PROJECT_ID", "ENTER YOUR WATSONX PROJECT ID"),
        params=parameters,
        chat=True
    )
    response = watsonx_llm.invoke(prompt)
    return re.sub(r"\*\*([^*]+)\*\*", r"\1", response)

@dataclass
class ResponseMetrics:
    """Class to store response and its metrics"""
    response: str
    processing_time: float
    start_time: datetime
    end_time: datetime
    token_count: int = 0
    cost_per_token: float = 0.0
    conciseness_score: float = 0.0

Utility functions for metrics calculation

These essential utility functions handle token counting, response evaluation, and cost calculations for our testing framework.

def count_tokens(text: str) -> int:
    """
    Simple token count estimation.
    """
    return int(len(text.split())*0.75)

def calculate_conciseness_score(response: str) -> float:
    """
    Calculate conciseness score based on response length and content.
    """
    words = response.split()
    ideal_length = 50  # adjust based on your needs
    length_score = 1 - min(abs(len(words) - ideal_length) / ideal_length, 1)
    return length_score

def get_cost_per_1k_token(model: str) -> float:
    """
    Get cost per token for different models.
    """
    cost_mapping = {
        'meta-llama/llama-3-1-70b-instruct': 0.0018,
        'meta-llama/llama-3-70b-instruct': 0.0010,
        'meta-llama/llama-3-8b-instruct': 0.0006,
        'mistralai/mistral-large': 0.01,
        'mistralai/mixtral-8x7b-instruct-v01': 0.0006,
        'ibm/granite-3-8b-instruct': 0.0002
    }
    return cost_mapping.get(model)

Parallel response generation system

This section implements the core parallel processing functionality that allows us to test multiple model-prompt combinations simultaneously, improving efficiency.

def generate_responses_parallel(prompt_list: List[str], model_list: List[str]) -> Dict[str, Dict[str, ResponseMetrics]]:
    """
    Generate responses for multiple prompts and models in parallel using ThreadPoolExecutor.
    """
    results = {}
    total_start_time = time.time()

    def process_single_combination(prompt: str, model: str) -> tuple:
        """Helper function to process a single prompt-model combination"""
        start_time = datetime.now()
        start_processing = time.time()

        try:
            response = generate_response(prompt, model)
            print(f'Model: {model}')
            print(f'Prompt: {prompt}')
            print(f'Response: {response}')
            processing_time = time.time() - start_processing
            end_time = datetime.now()

            # Calculate additional metrics
            token_count = count_tokens(prompt+response)
            cost_per_1k_token = get_cost_per_1k_token(model)
            conciseness_score = calculate_conciseness_score(response)

            metrics = ResponseMetrics(
                response=response,
                processing_time=processing_time,
                start_time=start_time,
                end_time=end_time,
                token_count=token_count,
                cost_per_token=cost_per_1k_token,
                conciseness_score=conciseness_score
            )
            return (prompt, model, metrics)
        except Exception as e:
            processing_time = time.time() - start_processing
            end_time = datetime.now()

            metrics = ResponseMetrics(
                response=f"Error: {str(e)}",
                processing_time=processing_time,
                start_time=start_time,
                end_time=end_time
            )

            return (prompt, model, metrics)

    # Create a list of all prompt-model combinations
    combinations = [(prompt, model)
                   for prompt in prompt_list
                   for model in model_list]

    # Initialize results dictionary
    results = {prompt: {} for prompt in prompt_list}

    # Use ThreadPoolExecutor for parallel processing
    with concurrent.futures.ThreadPoolExecutor(max_workers=min(len(combinations), 10)) as executor:
        future_to_combo = {
            executor.submit(process_single_combination, prompt, model): (prompt, model)
            for prompt, model in combinations
        }

        for future in concurrent.futures.as_completed(future_to_combo):
            prompt, model = future_to_combo[future]
            try:
                prompt_result, model_result, metrics = future.result()
                results[prompt_result][model_result] = metrics
            except Exception as e:
                end_time = datetime.now()
                results[prompt][model] = ResponseMetrics(
                    response=f"Error in processing: {str(e)}",
                    processing_time=0.0,
                    start_time=end_time,
                    end_time=end_time
                )
    total_time = time.time() - total_start_time
    print(f"Total processing time: {total_time:.2f} seconds")
    return results

Results analysis and reporting

This section handles the detailed analysis and presentation of our test results, providing both prompt-specific and aggregate statistics.

def print_detailed_statistics(results):
    """
    Print both per-prompt and aggregate statistics with detailed latency analysis.
    """
    # Per-prompt statistics
    print("\nPer-Prompt Statistics:")
    print("=" * 80)

    for prompt in results:
        prompt_latencies = []
        prompt_costs = []
        prompt_conciseness = []
        model_metrics = {}

        # Collect metrics for this prompt
        for model, metrics in results[prompt].items():
            prompt_latencies.append(metrics.processing_time)
            prompt_costs.append((metrics.token_count * metrics.cost_per_token))
            prompt_conciseness.append(metrics.conciseness_score)
            model_metrics[model] = metrics.processing_time

        # Print prompt-specific statistics
        print(f"\nPrompt: {prompt[:100]}...")  # Show first 100 chars of prompt
        print(f"Latency Statistics:")
        print(f"  Average Latency: {np.mean(prompt_latencies):.2f} seconds")
        print(f"  Best Latency: {min(prompt_latencies):.2f} seconds")
        print(f"     Model: {min(model_metrics.items(), key=lambda x: x[1])[0]}")
        print(f"  Worst Latency: {max(prompt_latencies):.2f} seconds")
        print(f"     Model: {max(model_metrics.items(), key=lambda x: x[1])[0]}")

        print(f"\nCost and Conciseness:")
        print(f"  Best Cost: {min(prompt_costs):.5f}")
        print(f"     Model: {min(model_metrics.items(), key=lambda x: x[1])[0]}")
        print(f"  Worst Cost: {max(prompt_costs):.5f}")
        print(f"     Model: {max(model_metrics.items(), key=lambda x: x[1])[0]}")
        print(f"  Average Conciseness Score: {np.mean(prompt_conciseness):.2f}")
        print("-" * 80)

    all_model_metrics = {}
    for prompt in results:
        for model, metrics in results[prompt].items():
            if model not in all_model_metrics:
                all_model_metrics[model] = {'latency':[], 'cost':[]}

            all_model_metrics[model]['latency'].append(metrics.processing_time)
            all_model_metrics[model]['cost'].append(metrics.token_count * metrics.cost_per_token)

    print("\nOverall Aggregate Statistics:")
    print("\nPer-Model Performance:")
    for model in all_model_metrics:
        model_metrics = all_model_metrics[model]
        print(f"\n  {model}:")
        print(f"    Average Latency: {np.mean(model_metrics['latency']):.2f} seconds")
        print(f"    Average Cost: {np.mean(model_metrics['cost']):.2f}")

Test execution configuration

This final section sets up our test cases and executes the parallel testing framework with specific models and prompts.

model_list = [
    'ibm/granite-3-8b-instruct',
    'meta-llama/llama-3-8b-instruct',
    'mistralai/mistral-large'
]
prompt1 = "What is the capital of France?"
prompt2 = '''You are tasked with reviewing system engineering requirements according to INCOSE guidelines. For the given requirement statement,
your objective is to list the specific INCOSE rules that are failing, along with brief explanations of why they are failing.
The output should be formatted as a JSON array where each item follows this pattern:

- Rule number: A reference to the INCOSE rule that is failing (e.g., "R1 - Structured Statements")
The input is as follows:

Input requirement: The system should be fast.
Output:
'''
prompt_list = [prompt1, prompt2]

# Generate responses
results = generate_responses_parallel(prompt_list, model_list)

# Print detailed statistics
print_detailed_statistics(results)

Future enhancements

To improve the LLM comparison framework, you can streamline the evaluation logic and add more targeted metrics. For example, metrics such as faithfulness (how accurately the model reflects the input query) and completeness (how thoroughly it addresses the task) enhance analysis depth.

Additionally, you can introduce an LLM-as-a-judge layer. This evaluator model analyzes reports generated from multiple LLMs and recommends the optimal model for specific use cases.

Conclusion

Comparing outputs from multiple LLMs is crucial for selecting the right model for specific tasks. With the growing variety of open-source models such as Llama, Granite, and Mistral, systematic comparison ensures that you maximize performance, minimize biases, and optimize costs.

By using a structured, automated framework, you can efficiently evaluate models on key metrics and streamline the decision-making process.

Products

Languages

Technologies

Events

External Resources

Champions

Comparing LLMs for optimizing cost and response quality