Personal finance area is becoming increasingly difficult to monitor and manage due to the flood of products. It is hard to see anyone these days without multiple bank accounts, credit cards, insurances, stocks, homes or bonds. There are a huge number of transactions generated as a result of this every month. These transactions need to be monitored and audited on a regular basis. It is difficult to get all information about a transaction from one source. There is a need to correlate information from multiple sources to get the complete picture about a transaction.

Let us take the example of credit card transactions. How many times have we looked at a transaction and thought – What is this and where did I spend this amount? All the people I have spoken to agree that this is a frequent occurrence.
Some of the reasons are listed below.

  • Instead of the merchant name, the payment gateway name appears on many transactions.
  • The merchant name that appears on transactions is different and not the same as the name of the retail store or online portal.
  • Need more information to recollect.
We all know it is a lot of effort to call the bank and get more details on a transaction. Is there a solution to this issue?

Every transaction has a date, amount and a description. In addition to this, we could have received an SMS or an e-mail related to the transaction. Suppose, we purchase an item on Amazon, we get an SMS and e-mail on the order confirmation. If we correlate all this information, we not only know we transacted on Amazon but we can know what we bought. This helps us recollect the transaction and save effort.
In this blog, let us explore the use of the Engineering Insights code pattern (Code repository) to correlate information across various sources for credit card transactions.

Let us look at a sample of actual data that we have from various sources for credit card transactions.
The below sample data from an actual credit card statement contains the below:
  • A date field
  • A description field that has the merchant name
  • Amount spent
Date Description Amount
2018-01-02 PAYU – BIG BASKET PG GURGAON 290.00

The below sample mail data contains information about the purchases:
  • Date
  • From
  • Order details in the mail text

Date From E-mail text
2018-01-02 Your Orders | Your Account | Order Confirmation. Order #XXX. Hello XXXX, Thank you for your order. We’ll send a confirmation when your order ships. Arriving: Friday, November 3 Your shipping speed: One-Day Delivery at Rs. 100. FREE with Prime Your Orders Your order will be sent to:—– Order Details Order #XXXX Placed on Tuesday, January 2, 2018 Harry Potter and the Goblet of Fire (Harry Potter 4) J.K. Rowling; Paperback Sold by Amazing Buy. Rs.380.00. Item Subtotal: Rs.380.00. Shipping & Handling: Rs.0.00. Order Total: Rs.380.00. Need to make changes to your order? Visit our Help Page for more information. Some products have a limited quantity available for purchase. Please see the product’s Detail Page for the available quantity. Any orders which exceed this quantity will be automatically canceled. We hope to see you again soon. Customers Who Bought Items in Your Order Also Bought Harry Potter and the Chamber of Secrets (Harry Potter 2) Harry Potter and the Chamber of Secrets Rs. 214.00. Harry Potter and the Philosopher’s Stone Harry Potter and the Philosopher’s Stone Rs. 207.00 Harry Potter and the Cursed Child: Parts I & II Harry Potter and the Cursed Child: Parts I & II Rs. 494.00 This email was sent from a notification-only address that cannot accept incoming email. Please do not reply to this message.
2018-01-02 Thank you for your order at Order No: XXXX. Your order will be delivered to this address:— . Delivery slot: Wed 03 Jan 2018 between 09:30 am and 12:00 pm. Sl No. Item Details Qty. Unit Price Sub Total Savings DRY ITEMS Foodgrains, Oil & Masala 1 24 Mantra Organic Flour – Ragi 500 gm Pouch 2 Rs. 60.00 Rs. 120.00 2 Organic Tattva Flour – Ragi 500 gm Pouch 2 Rs. 70.00 Rs. 140.00 Sub Total: Rs. 260.00 Delivery Charges: Rs. 30.00 Final Total: Rs. 290.00 Happy Shopping,bigbasket Team Note: To comply with GST, we have made a few process changes . Please visit to know more. Order Your Groceries On The Go! Get Social With Us. In case you need any assistance please do not hesitate to call our customer service on 1860 123 1000 07:00 am to 10:00 pm on all days or email us at All calls to our customer support number will be recorded for internal training and quality purposes.

The sample SMS data contains the below information:
  • Date
  • SMS text name
  • From
Date SMS Text From
2018-01-02 A charge of INR 290.00 has been initiated on your American Express Card No. *** at PAYU – BIG BASKET PG on 02/01/2018 at 03:16PM IST. AM-MYAMEX
2018-01-02 A charge of INR 380.00 has been initiated on your American Express card No. *** at AMAZON ONLINE DISTRIBUTION SERVICES PL on 02/01/2018 at 08:29AM IST. AM-MYAMEX

Data preparation
The data has to be prepared to enable the Engineering Insights code pattern to provide the best results. First, the date and amount field are good candidates for correlation.
We notice certain issues here:
  • This information exists outside the description field for the credit card transactions.
  • The format of the date is different in the mail and sms.
The above issues can be addressed in the following way:
  • Append the date and amount information to the Description field of the credit card transaction
  • Example: FRESH N FIT HOSPITALITI XXX 2018-01-02 00:00:00 165.0.
  • Append the date information to the SMS
  • Example: A charge of INR 165.00 has been initiated on your American Express Card No. ***** at SUBWAY on 02/01/2018 at 05:54PM IST. 2018-01-02 00:00:00.
  • Append the date information to the mail
  • Example: Hi ——, Thanks for using Swiggy! Your order from Chai Point has been delivered. Looking forward to serving you again. Order No: #—— Restaurant: Chai Point 2017-12-30 00:00:00.
Now the date format is consistent across all data sources. There is no need to specify different regular expressions for extracting dates with different date formats.
The description field for all data sources have all the available information. This data preparation was found to be sufficient to correlate the credit card transactions, smses and mails in the current scenario using the “Engineering Insights” code pattern.

We used a month of data consisting of 19 credit card transactions, 46 SMSes and 26 mails. The description field data from all data sources was fed to the Engineering Insights code pattern algorithm. The algorithm returned valuable correlations which can be seen below:
  • 1. Different name of merchant on credit card transaction
  • The transaction “FRESH N FIT HOSPITALITI XXX” was mapped to the below SMS – A charge of INR 165.00 has been initiated on your American Express Card No. *** at SUBWAY on 02/01/2018 at 05:54PM IST. This correlation made it clear that the amount was spent at SUBWAY. This was not clear from the credit card statement.
  • 2. Payment gateway name instead of merchant name on the credit card statement
  • The transaction RAZORPAY RETAIL PG BANGALORE was mapped to the mail – “Hi —-, Thanks for using Swiggy! Your order from Chai Point has been delivered. Looking forward to serving you again. Order No: #——— Restaurant: Chai Point” It becomes clear that this transaction was made at Chai Point.
  • 3. More details for a transaction
  • The transaction “AMAZON SELLER SERVICES BANGALORE” indicates that a purchase was made at Amazon. We might want to know what we ordered.
    This transaction maps to the e-mail – “Amazon Your Orders | Your Account | Shipping Confirmation Order #——– , We thought you’d like to know that we’ve dispatched your item(s). Your order is on the way. If you need to return an item from this shipment or manage other orders, please visit Your Orders on Arriving: Monday, November 6 Track your package Print invoice Your package was sent to: ———.Your package is being shipped by Amazon Transportation Services and the tracking number is —-. Please note that a signature may be required for the delivery of the package. Shipment Details The Matchless Wits: 3 in 1 (Amar Chitra Katha) Sold by UBSPD Rs.117.00 Item Subtotal: Rs.117.00 Shipping & Handling: Rs.0.00 Shipment Total: Rs.117.00 Paid by Amex: Rs.117.00 Track your order with the Amazon App. If you need further assistance with your order, please visit Customer Service. We hope to see you again soon!”
    The order details can be fetched based on the mapping.
A graph can be built based on all the mappings and stored into the graph database. A credit card transaction can be mapped to a SMS and E-mail.
For a credit card transaction, the graph database can be queried to get more information on a transaction based on the linked SMS or E-mail.

Practical considerations for solution implementation
This solution requires the access to credit card statements, SMSes and e-mails of Users. This requirement introduces practical difficulties for implementation.
1. Credit card statements
There are two ways to obtain the credit card statements for the solution:
  • The statements can be manually uploaded. This is an additional effort for the user.
  • The statements can be accessed from online portals of credit card companies. This requires the username and passwords for the credit card accounts to download the statements. This introduces security concerns that need to be addressed.
2. SMS
IPhone does not allow applications to access SMS. Hence, the SMS data cannot be automatically obtained for IPhones. For Android users, an Android app can access the SMSes and upload them for use by the solution. But, the users cannot restrict access to specific SMSes for applications which could be a security concern for some users.
3. E-mail
The access to e-mails can pose lots of security challenges. If it is a corporate e-mail account, it is not possible for an application to get access to the e-mails. If it is a public e-mail account like say gmail, the access to the mails can be got using API keys or OAuth. This introduces security concerns for users as the application gets control of the entire e-mail account. A workaround is forwarding of e-mails with filters to a common email account used by the solution. This requires a setup by the user on the e-mail account.
If the concerns expressed above do not matter or are addressed separately, the solution can provide control with convenience over personal finance matters.

We can use “Engineering Insights” code pattern to correlate information across other personal finance products like stocks, mutual funds, bank accounts and more. The solution can be further enhanced by integrating with IBM Watson services providing a mechanism to interact with the solution using natural language. For more details on the “Engineering Insights” pattern and usage, please visit the github repo at the URL – (

Join The Discussion

Your email address will not be published. Required fields are marked *