IBM Developer

Tutorial

Improve AI agent reliability with harness engineering and OpenAPI tool contracts

A practical introduction to AI agent harness engineering using IBM Bob, OpenAPI tool contracts, schema constraints, and API call logs to prevent invented values and improve AI agent reliability in watsonx Orchestrate

By Ramy Amer

When an AI agent behaves incorrectly, the first instinct is often to change the prompt. However, the prompt might not be the problem. An AI agent uses tools to perform actions, and each tool definition specifies parameter names, data types, accepted values, and instructions for using the tool. The agent reads this information on every turn. If a tool definition requires an account number but does not restrict the accepted values, the agent might invent an account number.

In watsonx Orchestrate, agents use tools imported from OpenAPI specifications. The OpenAPI specification acts as a tool contract that defines what the agent can send to the application programming interface (API). A tool contract that accepts any string can allow the agent to invent values, while a contract with a defined list of valid options limits the agent to those options.

In this tutorial, IBM Bob creates the backend service, OpenAPI specifications, and test setup. You will compare two agents that use the same model, description, and instructions but different tool contracts. You will send the same four requests to both agents and review the API call logs to see the values that each agent sends. You will learn why one agent succeeds while the other invents values and fails. You will also learn how to design a tool contract that prevents invalid API calls and how to verify an agent's actual actions from the call logs.

You also learn why tool contracts provide more control over agent behavior than prompts, how to identify common tool design problems in OpenAPI specifications, and how to improve a tool contract by removing unnecessary parameters, constraining accepted values, routing requests, simplifying definitions, and handling errors. You also learn how to use AGENTS.md to preserve requirements across multiple prompts in a Bob session.

Architecture for AI agent harness engineering demonstration

The tutorial uses a retail banking scenario to demonstrate AI agent harness engineering in which a customer asks an agent to transfer money between their accounts.

Architecture diagram: mock banking API, two OpenAPI tool contracts, two watsonx Orchestrate agents, and the API call log.

The agent uses a transfer tool to complete the request. The mock banking API and supporting files run on your local machine, while the two agents run in watsonx Orchestrate. Both agents call the same mock banking API through the same public tunnel. The only difference between the agents is the OpenAPI tool contract that defines the transfer tool.

The demonstration uses the following components:

  • Mock banking API (mock_api/main.py): The FastAPI service contains five accounts and enforces the banking rules. The API blocks transfers from closed or locked accounts and requires approval for transfers above 10,000 AUD. The service provides a permissive endpoint and a guarded endpoint that use the same business logic. The service logs every request before validating the request.

  • OpenAPI specifications (specs/transfer-v1.yaml and specs/transfer-v2.yaml): The two OpenAPI specifications define the tool contracts. The first specification accepts four free-text parameters without restrictions. The second specification accepts two parameters and restricts the account and transfer amount to valid values.

  • watsonx Orchestrate agents (transfer_agent_v1 and transfer_agent_v2): The two agents use the same model, description, and instructions. Each agent uses one transfer tool that is imported from a different OpenAPI specification.

  • Public tunnel: ngrok or cloudflared provides a public URL for the mock banking API running on your local machine. The public URL allows the agents in watsonx Orchestrate to call the local API.

  • API call log (logs/calls.jsonl): The call log records every request sent to the mock banking API, including the arguments supplied by the agent and the resulting status. You will use the call log to compare the behavior of the two agents.

Demonstration workflow

You send a request through Bob to an agent in the watsonx Orchestrate chat panel. The agent uses the tool contract to select a tool and determine which arguments to send. The tool call passes through the public tunnel to the mock banking API. The mock banking API logs and validates the request, then completes or rejects the transfer. The agent receives the API response and returns a response in the chat panel.

Each agent uses a tool that is imported from a different OpenAPI specification. Both agents use the same default model, description, and instructions. Because watsonx Orchestrate runs in the cloud and the mock banking API runs on your local machine, the OpenAPI specifications use the public tunnel URL to connect watsonx Orchestrate to the mock banking API.

The mock banking API provides the evidence for comparing agent behavior. The API logs every request before validation, including the request body, response status, and forwarding headers. The log records both successful and rejected requests. You can review the API log to see the exact values that each agent sent. The API log provides server-side evidence of agent behavior instead of relying on the agent response.

Prerequisites

Before you begin, make sure that you have the following:

Important guidelines

  • Use the file names, seed state, field names, and enum values exactly as provided. Do not rename or replace any values.
  • Copy the agent names, descriptions, and instructions exactly into AGENTS.md. Verify that Bob reproduces the content without any changes.

Steps

You do not write the code yourself. Bob builds the backend service, both tool contracts, and the test setup from the prompt files in the GitHub repository. You send the prompts to Bob one step at a time and review the output from Bob to understand why the first tool contract allows invalid agent behavior and how the second tool contract improves agent behavior. Specification A is intentionally permissive so you can clearly compare the improvements in Specification B.

Step 1. Set up IBM Bob and the watsonx Orchestrate environment

  1. Clone the GitHub repository. Run the following command in a terminal or IBM Bob.

    git clone https://github.com/IBM/oic-i-agentic-ai-tutorials.git
    
  2. Open the Harness_Engineering folder in the cloned repository. The folder contains separate prompt files and an AGENTS.md file, which you will include with the first prompt.

  3. Open IBM Bob and select the Harness_Engineering folder.

    IBM Bob with the Harness_Engineering folder selected.

  4. Sign in to watsonx Orchestrate and click Create agent. Select the option to launch IBM Bob.

    watsonx Orchestrate Create agent panel with the option to launch IBM Bob.

  5. Click Continue with Current Folder to use the Harness_Engineering folder. This action connects IBM Bob on your local machine to your watsonx Orchestrate environment.

    IBM Bob prompt to continue with the current folder.

  6. Enter the API key when Bob prompts for the key. Bob then connects to your watsonx Orchestrate environment.

    IBM Bob prompt for the watsonx Orchestrate API key.

  7. Click Explorer in Bob. Bob creates the watsonx Orchestrate ADK project structure, including agents/, tools/, connections/, knowledge-bases/, models/, toolkits/, workspace_config.yaml, .bob, .vscode, and venv.

    IBM Bob Explorer showing the watsonx Orchestrate ADK project structure.

  8. Run the following command in the terminal.

     orchestrate env list
    

    Confirm that the correct watsonx Orchestrate environment is active. The command lists the configured environments and identifies the active environment.

    Terminal output showing the list of configured watsonx Orchestrate environments.

  9. Run the following command to identify the default model.

     orchestrate models list
    

    Find the default model that is marked with a check mark and copy the exact model ID. Save the model ID for use later in the tutorial. Do not retype the model ID because an incorrect model ID can cause both agents to fail.

    Terminal output showing the list of available models with the default model marked.

Step 2. Create the project and the rules file

  1. Enter the following prompt in the Bob chat. Bob resolves the file path from the current working folder. Ensure that the GitHub repository is cloned into the project folder that you opened in Bob. If Bob cannot find a prompt file, copy the prompt file into the project folder and try again.

    Implement the spec @create_project_and_rules_file.md
    

    IBM Bob chat showing the prompt to create the project and rules file.

    Bob creates the AGENTS.md file and displays the file contents. The AGENTS.md file (contract file) defines the requirements that IBM Bob must follow throughout the tutorial.

    Compare the generated AGENTS.md file with the AGENTS.md file in the GitHub repository and confirm that all instructions, names, and enum values match exactly. Even a small change to an instruction or enum value can affect the agent behavior in the tutorial.

    Checkpoint 1. Verify the project structure

    Confirm that AGENTS.md exists in the project root and matches the source file in the GitHub repository exactly. Confirm that Bob created the mock_api/, specs/, eval/, and logs/ folders.

    The project also contains the watsonx Orchestrate scaffold: agents/, tools/, connections/, knowledge-bases/, models/, toolkits/, and workspace_config.yaml, along with .bob, .vscode, and venv. These files and folders are part of the existing watsonx Orchestrate project setup and were not created by Bob in this step.

Step 3. Build the mock banking API

The mock banking API in mock_api/main.py is shared by transfer_agent_v1 and transfer_agent_v2. Using the same backend logic for both agents ensures that the tool contract is the only difference in the comparison.

  1. Enter the following prompt in the Bob chat.

     Implement the spec @build_mock_backend.md
    

    Bob might request several approvals while creating the backend. Approve the requests to continue.

    IBM Bob chat showing the prompt to build the mock banking API.

    IBM Bob requesting approval to create the mock banking API files.

    The AGENTS.md file defines specific requirements to keep the comparison between the two agents consistent. The following table explains the problems that can occur when these requirements are not followed.

    Pinned detail What went wrong in our run without it
    Field names for both endpoints The guarded endpoint expected amount while Specification B sends amount_aud. Every valid v2 call was rejected, and the agent replied, "I am sorry, I did not understand your question."
    One shared account resolver The guarded endpoint looked up accounts by ID only, so a valid call with SAVINGS was rejected as unknown. Different rules for each endpoint would also make the comparison inconsistent.
    Destinations typed in the model With to_account typed as a plain string, any value passed validation. As a result, the remediation message did not run, and EVERYDAY could have been accepted as a destination.
    No 10,000 AUD cap in the model A model-level cap replaces the createPaymentApproval remediation with a generic range error.
    Override validation errors on the permissive endpoint FastAPI's default 422 response identifies the invalid field and explains the error, which adds guidance to the permissive agent's "bare errors only" condition.
    Log in middleware before validation Requests rejected by validation never reach the handler. Handler-level logging therefore misses the rejected calls that the experiment needs to count.
    Forwarding headers in the log Without forwarding headers, a test call from a local terminal is indistinguishable from an agent call.
  2. Bob does not start the mock banking API. Open a new terminal in VS Code by selecting Terminal > New Terminal.

    1. In Terminal 1, activate the virtual environment and start the mock banking API:

       source venv/bin/activate
       uvicorn mock_api.main:app --port 8080
      

      Wait for the Application startup complete message.

      Terminal showing the Application startup complete message for the mock banking API.

      If the uvicorn command is not found, confirm that the virtual environment is active. If FastAPI is not installed, run:

       pip install fastapi uvicorn
      
    2. Open Terminal 2 and run the following commands to test four API behaviors:

       curl -s -w "\nHTTP %{http_code}\n" -X POST localhost:8080/v1/transfer-guarded \
       -H "Content-Type: application/json" \
       -d '{"to_account":"EVERYDAY","amount_aud":200}'
      
       curl -s -w "\nHTTP %{http_code}\n" -X POST localhost:8080/v1/transfer-guarded \
       -H "Content-Type: application/json" \
       -d '{"to_account":"SAVINGS","amount_aud":25000}'
      
       curl -s -w "\nHTTP %{http_code}\n" -X POST localhost:8080/v1/transfer-guarded \
       -H "Content-Type: application/json" \
       -d '{"to_account":"SAVINGS","amount_aud":200}'
      
       curl -s -w "\nHTTP %{http_code}\n" -X POST localhost:8080/v1/transfer \
       -H "Content-Type: application/json" \
       -d '{"from_account":"OLD_SAVER","to_account":"SAVINGS","amount":100}'
      

      Terminal showing the four curl test results for the mock banking API.

      Expected API test results

      Test Expected result
      Invalid destination HTTP 422 with {"error":"to_account must be one of SAVINGS, OFFSET, TERM_DEPOSIT."}
      Over the limit HTTP 422 with an error pointing to createPaymentApproval.
      Valid transfer HTTP 200 with exactly four fields: status, destination, amount_aud, and new_balance_aud.
      Closed source account on the permissive endpoint HTTP 400 with {"error":"invalid request"} and no additional information.

      Note: A % at the end of a response in zsh indicates that the response does not end with a newline. The % is not part of the API response. If a test fails, specify Bob that test failed and include the response that you received.

  3. A successful transfer changes the account balances stored in memory. Restart the mock banking API after any test that transfers money so that each test starts with the account balances defined in AGENTS.md.

    To reset the account balances, stop the mock banking API by pressing Ctrl+C in the terminal where the API is running. Then, restart the mock banking API.

     uvicorn mock_api.main:app --port 8080
    

    The account balances return to the values defined in AGENTS.md. The public tunnel remains active and keeps the same URL.

Checkpoint 2. Verify the mock banking API

Confirm that the mock banking API starts successfully and that all four curl tests return the expected HTTP status and response. Verify that logs/calls.jsonl contains an entry for every request, including rejected requests. Restart the mock banking API to reset the account balances before continuing.

Terminal showing the logs/calls.jsonl file with entries for all four curl test requests.

Step 4. Create specification A with a weak tool contract

Specification A is intentionally weak to demonstrate a tool design anti-pattern. The tool contract includes the fields required for watsonx Orchestrate to import the tool, but the tool description does not explain when the agent should use the tool. The four parameters also accept unrestricted values, leaving the agent to determine valid account identifiers and other values.

  1. Enter the following prompt in the Bob chat.

    Implement the spec @write_spec_a_anti_pattern.md
    

    IBM Bob chat showing the prompt to create Specification A.

    Verify the generated Specification A file at specs/transfer-v1.yaml. The formatting and info values might differ, but the tool definition should look similar to the following example.

    openapi: 3.0.3
    info:
      title: ...
      version: ...
    paths:
      /v1/transfer:
        post:
          operationId: transferFunds
          summary: Transfers funds between accounts.
          description: Transfers funds between accounts.
          requestBody:
            content:
              application/json:
                schema:
                  type: object
                  properties:
                    from_account:
                      type: string
                    to_account:
                      type: string
                    amount:
                      type: number
                    currency:
                      type: string
    

    Specification A contains four unrestricted parameters and a vague tool description. As a result, the tool contract allows schema-valid requests with an unknown account, unsupported currency, negative amount, or source account that the customer does not own.

Checkpoint 3. Verify specification A

Confirm that specs/transfer-v1.yaml includes openapi, info, the vague summary and matching description, and four optional parameters without constraints. The specification must not include enums, value bounds, or guidance for invalid requests.

If Bob adds any constraints or guidance, ask Bob to remove the changes so that Specification A remains intentionally weak.

Step 5. Create and harden specification B

Specification B uses a more restrictive tool contract. First, Bob creates an initial version of the specification. Bob then identifies missing constraints and creates a hardened version. Comparing the two versions shows how the tool contract becomes more restrictive.

  1. Enter the following prompt in the Bob chat.

     Implement the spec @write_spec_b_mistake_proofed.md
    

    IBM Bob chat showing the prompt to create Specification B.

    Review the initial Specification B draft that Bob generates. Bob identifies the missing constraints before creating the hardened version.

    IBM Bob chat showing the initial Specification B draft before hardening.

    IBM Bob chat showing the hardened Specification B.

    The relevant part of the hardened specification should look similar to the following example:

     /v1/transfer-guarded:
       post:
         operationId: transferToOwnAccount
         summary: >
           Transfer funds from the signed-in user's everyday account to one of their
           linked accounts. Use for user-initiated transfers between the user's own
           accounts. Do NOT use for third-party payments (use createPayeePayment) or
           for scheduled or recurring transfers (use scheduleTransfer).
         description: >
           (identical to summary)
         requestBody:
           content:
             application/json:
               schema:
                 type: object
                 required:
                   - to_account
                   - amount_aud
                 properties:
                   to_account:
                     type: string
                     enum:
                       - SAVINGS
                       - OFFSET
                       - TERM_DEPOSIT
                     description: Destination account, from the user's linked accounts.
                   amount_aud:
                     type: number
                     minimum: 1
                     maximum: 10000
                     description: >
                       Amount in whole AUD. Transfers above 10000 require
                       createPaymentApproval.
         responses:
           '422':
             description: >
               Returns a remediation message the agent can act on, for example
               "to_account must be one of SAVINGS, OFFSET, TERM_DEPOSIT."
    

What changed: The agent no longer provides the source account or currency. The backend resolves the source account, and the API endpoint defines the currency. The remaining parameters are constrained to three valid destination accounts and a defined amount range. The tool contract now has two constrained parameters instead of four unrestricted parameters, reducing the chance of invalid values.

Checkpoint 4. Verify specification B

Confirm that IBM Bob paused after creating the first draft and continued after your approval. You should have two versions to compare: the first draft from the IBM Bob chat and the final specs/transfer-v2.yaml file.

Verify that specs/transfer-v2.yaml contains the following requirements:

  • operationId is exactly transferToOwnAccount.
  • Contains exactly two properties: to_account and amount_aud. Both properties are required.
  • to_account uses the enum [SAVINGS, OFFSET, TERM_DEPOSIT].
  • amount_aud has minimum: 1 and maximum: 10000.
  • The summary and description contain the exact required text, including the Do NOT use clause.
  • The successful response contains four fields.
  • Documents the 422 response.
  • Includes an openapi: 3.0.x declaration and an info block.
  • from_account and currency do not appear anywhere in the specification.

Formatting, line wrapping, info values, and the layout of the 200 response block can differ.

Step 6. Connect the mock banking API to watsonx Orchestrate

watsonx Orchestrate runs in the cloud, while the mock banking API runs on your local machine. A public tunnel provides a URL that allows watsonx Orchestrate to access the mock banking API.

For this step, open three terminal tabs in IBM Bob. Use Terminal 1 for the mock banking API, Terminal 2 for the public tunnel, and Terminal 3 to test the tunnel.

Terminal 1: Run the mock banking API

If the mock banking API is not already running, activate the virtual environment and start the API.

source venv/bin/activate
uvicorn mock_api.main:app --port 8080

Wait for the Application startup complete message.

Terminal 2: Start the public tunnel

If ngrok is not installed, install ngrok:

brew install ngrok

Terminal showing the ngrok installation command.

If ngrok requires authentication, copy the authentication token from your ngrok account and run the following command.

ngrok account page showing the authentication token.

ngrok config add-authtoken <your-token>

Start the ngrok tunnel:

ngrok http 8080

Copy the HTTPS forwarding URL displayed by ngrok. You will use this URL to connect watsonx Orchestrate to the mock banking API.

Terminal showing the ngrok tunnel running with the HTTPS forwarding URL.

Terminal 3: Test the public tunnel

Test the tunnel before adding the tunnel URL to the OpenAPI specifications. If the tunnel test fails, watsonx Orchestrate cannot access the mock banking API.

IBM Bob with three terminal tabs open for the mock banking API, the public tunnel, and the tunnel test.

  • In Terminal 3, replace YOUR-TUNNEL-URL with the HTTPS forwarding URL from ngrok. Run the following command to verify that the public tunnel can reach the mock banking API:

      curl -s -w "\nHTTP %{http_code}\n" -X POST \
        https://YOUR-TUNNEL-URL/v1/transfer-guarded \
        -H "Content-Type: application/json" \
        -d '{"to_account":"SAVINGS","amount_aud":200}'
    

    The request should return HTTP 200. The request should also appear in Terminal 1, where the mock banking API is running.

    Terminal showing the tunnel test returning HTTP 200 and the request appearing in the mock banking API terminal.

    The test transfers 200 AUD from the EVERYDAY account to the SAVINGS account. Because the mock banking API stores account balances in memory, restart the mock banking API to restore the original balances. In Terminal 1, press Ctrl+C, and then run:

      uvicorn mock_api.main:app --port 8080
    

    Restarting the mock banking API does not change the tunnel URL.

  • Replace <PASTE TUNNEL URL> with the HTTPS forwarding URL. Then, enter the following prompt in the Bob chat.

    The mock API is running and exposed at <PASTE TUNNEL URL>.
    
    Add a servers block with that URL to both specs/transfer-v1.yaml and
    specs/transfer-v2.yaml, identically. Change nothing else. Show me the diff.
    

    IBM Bob chat showing the diff after adding the servers block to both OpenAPI specifications.

Notes:

  • Keep Terminal 1 and Terminal 2 open until you complete the tutorial. Bob cannot keep long-running processes active. If either process stops, the watsonx Orchestrate agents cannot access the mock banking API.
  • A free ngrok tunnel URL can change when the tunnel restarts. If the tunnel URL changes, update the servers block in both OpenAPI specifications and update both tools in watsonx Orchestrate with the new URL.

Checkpoint 5. Verify the tunnel connection

Confirm that the curl request through the tunnel returns HTTP 200. Verify that both OpenAPI specifications contain the same servers block with the current tunnel URL and that no other content has changed. Confirm that the mock banking API was restarted after the test transfer and that both the API server and ngrok tunnel are still running.

Step 7. Deploy both agents to watsonx Orchestrate

In this step, use IBM Bob to import the two OpenAPI tools and create both agents in watsonx Orchestrate. IBM Bob displays the commands and waits for your approval before creating the tools and agents.

  1. Enter the following prompt in the IBM Bob chat. Replace the model ID placeholder with the exact model ID that you copied from orchestrate models list.

     Follow AGENTS.md, especially rules 2, 4, 8 and 9.
    
     Deploy both agents to my watsonx Orchestrate instance, in this order:
    
     1. Import specs/transfer-v1.yaml and specs/transfer-v2.yaml as OpenAPI tools.
     2. List the tools in Orchestrate and confirm exactly one transferFunds and one
     transferToOwnAccount exist.
     3. Create two agents:
     transfer_agent_v1: tool: transferFunds
     transfer_agent_v2: tool: transferToOwnAccount
    
     Both get the pinned description and the pinned instruction block from AGENTS.md,
     verbatim, and this model ID for both:
     <PASTE THE EXACT ID FROM orchestrate models list>
    
     Before you create anything, print the exact calls or commands you intend to run and
     wait for my confirmation.
    
     After both agents exist:
     - read back each agent's model, description, instructions and tools as a table
     - fetch each agent's instruction block from Orchestrate separately and print a SHA-256
     hash of each; do not hash one string and apply it to both
    
     Then stop.
    

    Bob displays the import and agent creation commands and waits for your approval. Before approving the commands, verify the following details:

    • The agent names are transfer_agent_v1 and transfer_agent_v2.
    • Each agent has one tool.
    • The model ID matches the exact ID from orchestrate models list.
    • The agent description and instructions match the content in AGENTS.md exactly.

      IBM Bob chat showing the deployment commands for the two agents before approval.

      IBM Bob chat showing the agent creation confirmation with model, description, and tool details.

      After you verify the commands, tell Bob to continue with the deployment.

  2. After Bob creates both agents, verify the agent configurations. The two agents should use the same model, description, and instructions. Only the assigned tool should differ.

    Field transfer_agent_v1 transfer_agent_v2
    Model groq/openai/gpt-oss-120b Identical
    Description Helps a retail banking customer move money between their own accounts. Identical
    Instructions You are a retail banking assistant for Alex Chen... Identical
    Tool transferFunds transferToOwnAccount

    watsonx Orchestrate showing both agents configured with the same model and different tools.

  3. Test both agents first with a simple greeting and then with a transfer request. Enter the following prompt in the Bob chat.

     Two smoke tests, then stop:
    
     1. Send "Hello" to transfer_agent_v1 and to transfer_agent_v2. Report each response.
    
     2. Send "Transfer 200 to my savings" to each agent. When the agent asks for confirmation,
       reply "yes, proceed" in the same conversation. For each agent, show the final
       response and the tool-call arguments from the Orchestrate trace, if available, and
       the matching lines from logs/calls.jsonl.
    
     If you cannot retrieve tool-call arguments from either source, stop and tell me.
    
    • Monitor Terminal 1 while the smoke tests run. Requests from watsonx Orchestrate appear from external IP addresses. Requests that you send locally with curl appear as 127.0.0.1.
    • The permissive endpoint might return 400 Bad Request when transfer_agent_v1 sends an invalid value. This response means that the mock banking API correctly rejected the invalid request.

      IBM Bob chat showing the smoke test prompts sent to both agents.

      The following screen shows the output from the first smoke test.

      IBM Bob chat showing the Hello response from transfer_agent_v1.

      IBM Bob chat showing the transfer response from transfer_agent_v2.

  4. Verify the deployment before continuing. Do not continue if the tool-call arguments are not available in the watsonx Orchestrate trace or logs/calls.jsonl. The tool-call arguments provide the evidence needed to compare the two agents.

    IBM Bob chat showing the tool-call arguments from logs/calls.jsonl for the smoke tests.

    Before continuing, verify the following:

    • Both OpenAPI specifications are imported, with exactly one transferFunds tool and one transferToOwnAccount tool.
    • Both agents exist with the names transfer_agent_v1 and transfer_agent_v2.
    • Both agents use the same model, description, and instructions. Only the tools differ.
    • Both agents respond correctly to the Hello test.
    • Tool-call arguments are available for both transfer tests.
  5. The smoke tests changed the account balances stored in memory. Restart the mock banking API before continuing with the next step so that both agents start with the account balances defined in AGENTS.md.

    In the uvicorn terminal, press Ctrl+C, and then restart the mock banking API.

     uvicorn mock_api.main:app --port 8080
    

    If port 8080 is already in use, stop the existing process.

     kill $(lsof -ti:8080)
    

    Keep the ngrok tunnel running. Restarting the mock banking API does not change the tunnel URL.

Step 8. Compare both tool contracts with the same scenarios

Test transfer_agent_v1 and transfer_agent_v2 with the same four scenarios. Use logs/calls.jsonl to compare the arguments each agent sends and the HTTP status returned by the mock banking API.

  1. Enter the following prompt in the Bob chat.

     Run four scenarios against transfer_agent_v1, then the same four against
     transfer_agent_v2. Each scenario in its own conversation:
    
     1. "Transfer 200 to my savings"
     2. "Move some money to my savings" (if it asks how much, reply "200")
     3. "Pay my landlord 600"
     4. "Transfer 2000 from my term deposit to savings"
    
     When an agent asks for confirmation, reply "yes, proceed" in the same conversation.
    
     For each scenario and agent, show me the final response and the matching lines from
     logs/calls.jsonl: path, arguments and status. Quote the arguments from the log;
     do not summarise them.
    

    IBM Bob chat showing the four scenario prompts sent to both agents.

    Review the API call log

    Use logs/calls.jsonl as the source of evidence for each test. An agent might report a successful transfer even when the mock banking API rejected the request. Compare the exact tool-call arguments and HTTP status in logs/calls.jsonl to determine what each agent actually sent to the API.

    Results

    Both agents used the same groq/openai/gpt-oss-120b model, description, and instructions. The only difference was the tool contract.

    In the example test results, transfer_agent_v1 made 14 API calls, and the mock banking API rejected all 14 calls. The agent generated different values for the source account, including checking, checking_account, Checking, checking-1, and primary_checking. Some requests omitted currency or from_account. The results show that an unrestricted tool contract can cause the agent to try different parameter values after a failed request.

    transfer_agent_v2 made three API calls. Each call sent {"to_account": "SAVINGS", "amount_aud": 200}, and the mock banking API accepted all three calls with HTTP 200. Specification B does not require the agent to provide the source account, which removes the parameter that caused the failures in Specification A.

    Overall, the mock banking API accepted 0 of 14 calls from transfer_agent_v1 and 3 of 3 calls from transfer_agent_v2.

    logs/calls.jsonl showing 14 rejected calls from transfer_agent_v1.

    logs/calls.jsonl showing 3 accepted calls from transfer_agent_v2.

    Every call from transfer_agent_v1 used invented values and was rejected. Every call from transfer_agent_v2 used valid values and was accepted. Both agents used the same model and instructions.

    For the successful transfer_agent_v2 calls, the agent sent:

     {"amount_aud": 200, "to_account": "SAVINGS"}
    

    Agent test walkthrough

    Case transfer_agent_v1 permissive transfer_agent_v2 mistake-proofed
    "Move some money to my savings" → "200" The agent asked for the amount and source account. The agent then made three calls with invented values. For example, one call used {"amount": 200, "currency": "USD", "from_account": "checking", "to_account": "savings"}. All three calls returned HTTP 400. The agent then asked for account identifiers. The agent asked for the amount and then made one call with {"amount_aud": 200, "to_account": "SAVINGS"}. The API returned HTTP 200, and the transfer completed.
    "Transfer 500" → "to my savings" The agent made one call with checking, savings, and USD. The API returned HTTP 400, and the transfer did not complete. The agent did not make a tool call because the conversation failed at the follow-up turn.
    "Put a bit aside for later" → “200 to my savings” The agent made two calls with invented values. Both calls returned HTTP 400. The agent then told the user, "The $200 has been moved." No money was transferred. The agent made one call with {"amount_aud": 200, "to_account": "SAVINGS"}. The API returned HTTP 200, and the transfer completed.

Tradeoffs of tool contract constraints: Constraining a tool contract has two important tradeoffs. A detailed tool description can help the agent select the correct tool, but a long generated tool name can make the tool harder for the model to select again later in a conversation. Removing a parameter also removes the agent's ability to provide that value. Removing a parameter works well when the value should never change, but it can limit the tool when the value needs to vary.

watsonx Orchestrate tools list showing the truncated v2 tool name.

In the watsonx Orchestrate tools list, the v1 tool has a short name. The v2 tool uses the full routing description as the tool name, so watsonx Orchestrate truncates the name. On this platform, the tool description is used as the tool identifier.

Checkpoint 6. Verify the agent comparison

Confirm that you tested the same four scenarios with both agents. For each scenario, verify that you recorded the agent response and the exact tool-call arguments from logs/calls.jsonl.

Step 9. Verify the acceptance criteria

Use Bob to verify the acceptance criteria in AGENTS.md. Bob must check the current project state and provide evidence for each result.

  1. Enter the following prompt in the Bob chat.

    Follow AGENTS.md.
    
    Verify the global acceptance criteria in AGENTS.md one by one. For each, state PASS or
    FAIL and the evidence you checked NOW: a file path, a command output, or a hash. Do not
    mark anything PASS because you believe you did it earlier.
    
    Run the smoke tests in mock_api/README.md and quote their output.
    
    Then list, in one line each, anything in this project that a reader reproducing it
    would be likely to get differently, and why.
    
  2. Review the results from Bob. Confirm that each acceptance criterion includes a PASS or FAIL result and supporting evidence from the current project.

    IBM Bob chat showing the acceptance criteria verification results.

  3. The smoke tests can change the account balances stored in memory. Restart the mock banking API after the smoke tests if you plan to run additional tests.

Checkpoint 7. Verify the acceptance criteria

Confirm that every acceptance criterion in AGENTS.md has a PASS result with supporting evidence collected during this step.

Summary

You did not write the code yourself. Bob created the code from the prompts in the repository. You compared two agents that differed in only one area: the tool contract.

The permissive tool contract allowed the agent to invent an account that did not exist, a currency the banking service did not support, and a source account the agent could not identify. The mistake-proofed tool contract removed or restricted these values, leaving fewer opportunities for the agent to invent values.

The key lesson is where the evidence came from. The mock banking API call log showed what each agent actually sent, rather than relying on what the agent reported. An agent can report success even when the backend rejects the request, and the API call log provides the evidence needed to verify the result.

IBM Bob and watsonx Orchestrate provided the components needed to test this behavior. The OpenAPI specifications, agents, AGENTS.md rules, and API call log together form the harness around the model. Designing this harness carefully can make agent behavior more reliable.

Acknowledgements

The author (Ramy Amer) deeply appreciates the support of Ela Dixit and Bindu Umesh for the guidance and expertise on reviewing and contributing to this tutorial.

This tutorial was produced as part of the IBM Open Innovation Community initiative.