Trade Assistant with Watson

With contributions by Micah Forster, Sai Gudimetla, and Jeff Powell.

Even with the best player insights and football expertise, fantasy football can be a losing proposition if a team does not have a good mix of players. The lack of depth at certain positions does not pair well with today’s football reality: prolific injuries to key players, COVID-19 game postponements, players opting out of the season, moving of bye weeks, and the change of rules for the injured reserve designation. Team managers can quickly discover that their team has a low score ceiling even if they start their best active players. A loss of fantasy football hope can quickly impair future decisions, which can fade your playoff hopes.

Trade Assistant with Watson analyzes combinations of players and their related data to objectively find trades for your team. Teams have the opportunity to increase their winning potential by accepting artificial intelligence (AI)-generated trades. The flow of player assets between teams is facilitated by a diversity of trade packages that balance fairness, realism, and value in different ways. Trades are personalized on demand. They consider your team’s roster, league rules, and competition.

The technology behind Trade Assistant with Watson uses a combination of AI, optimization techniques, and a hybrid cloud to deliver high-quality trades at speed. In this first part of a three-part series, let’s find out how.

Architecture

The system has supporting components that run in either batch or real-time mode. As seen in Figure 1, a continuous process running in the Trade Assistant engine Python application precomputes broad variables that estimate a fair market valuation for each football player. The application pulls data from the Player Insights with Watson Db2® database and joins it with ESPN Fantasy Football statistics. The resulting player valuations are stored within the Player Insights Db2 database. After completion of the job, the Trade Content Generator Node.js application is called through a get request.

A thread is started on the Node.js application that pulls the player market valuation data from Db2. The application converts the row-based data into JSON files and pushes the content to the IBM Cloud® Object Storage, which is purged. The application assembles both player-specific files and an overall valuation file to optimize algorithm performance. For example, we store each player’s boom probability, bust chance, position owned, position rank, binned rank, valuation, injury status, game status, team ID, and opponent team ID for each week in the overall JSON file. The content populates throughout the IBM Content Delivery Network and cascades through the ESPN Content Delivery Network. The real-time applications consume the broad market valuation file as an input into the Trade Assistant algorithm.

Trade Assistant with Watson system architecture Figure 1. Trade Assistant with Watson system architecture

A third application called the Algorithm Bench is a Node.js application and service that contains the core Trade Assistant optimization logic. The Algorithm Bench is used to test and develop the AI algorithms that are used in two places. The Node.js code is packaged as a library and imported into a Football Error Analysis Tool (FEAT) application. The web-based application is delivered through a browser so that ESPN and IBM Trade Assistant analysts can teach Watson good and bad trades. FEAT is written in the React framework that provides a modular architecture to flexibly build user experiences. The analysis completed through FEAT is implemented in the Algorithm Bench libraries.

Consumer traffic from the ESPN Fantasy Football mobile app requests trades from the Algorithm Bench Fastify app that’s deployed on 525 Red Hat OpenShift® PODs across three geographical regions. We are able to provide continuous availability with three active Red Hat OpenShift Kubernetes Service (ROKS) clusters while maintaining a subsecond response time. We surpassed our goal of providing trade recommendations under five seconds with a peak load of 1,000 requests a second. All of the consumer traffic is brokered by Global Traffic Management.

All of the ESPN league data that the Trade Assistant with Watson requires is passed into the algorithm through a post request. The trade valuation broad variables are periodically loaded by a scheduler into each of the 525 PODs’ memory by a get request. The process requests the player valuation JSON files from the ESPN Content Delivery Network that was created by the Trade Assistant Engine Python app and uploaded to IBM Cloud Object Storage by the Trade Content Generator Node.js app. The algorithm that creates player market valuations considers many factors that are used by the overall Trade Assistant system. Let’s find out which ones.

General Trade Assistant algorithm

Trade Assistant with Watson follows nine steps, shown in Figure 2, to produce trades that balance value and fairness.

Steps that generate trade packages Figure 2. The nine steps that generate trade packages

  1. Broad market values are for the top players in football within a continuously running batch job. Data that is produced by the Player Insights with Watson such as debiased boom and bust probabilities are joined with ESPN data. These pieces of information along with Watson Discovery and Natural Language Understanding are integral components of a player’s fair market value.

  2. Broad player cost components are calculated within the same job as step 1. For example, game projections and positional rankings are pulled from both ESPN APIs and the IBM Player Insights with Watson Db2 database to determine the opportunity cost of a team losing a player.

  3. Next, we begin creating personalized player market valuations by modifying broad player market valuations from step 1. The ESPN Fantasy Football team manager’s league rules, roster, and competition contribute to the new estimation.

  4. In the same process as step 3, personalized player costs are calculated from broad cost estimates from step 2. The cost measures the loss a team will suffer by losing the fantasy football player within a trade.

  5. Now, each team within a league is analyzed and summarized based on their roster.

  6. The results from step 5 are used to match teams together for trading based on offsetting each other’s strengths and weaknesses.

  7. Trade packages are generated by maximizing the return of player values and minimizing the cost of sending away players within a trade. The maximum cost a team can take is equal to a maximum cost their trading partner could sustain.

  8. Each of the trade packages are analyzed for fairness and value. The fairness metric considers both the benefit and realism for both sides of a proposed trade. The value metric provides a selfish view of the value you will receive regardless of your opponent’s point of view.

  9. Several analytic filters and business rules are applied to the trade packages to balance trade quality and volume. The remaining trades are viewable by the consumer within the ESPN Fantasy Football mobile apps.

Let’s go into more detail on how we produce trades.

Player market valuation

Each of the top 450 players in football based on percentage owned in all leagues including all of the kickers and defenses are included for market valuation. All of the players have a feature vector that is generated by retrieving data from the ESPN APIs and Player Insights with Watson.

After the initial player valuation data is retrieved, the player valuation vectors are updated with boom and bust ratios for each player. Next, a few of the players are filtered if they do not have a minimal rest of the season projection, which can mean that a player is out due to COVID-19 or for other dynamic situations such as game postponements.

Next, we determine derived statistics about each of the players. The full season projection valuation is calculated by taking the cumulative probability function of the normal distribution across all players for the current season.

At this point, the broad market value of a player is stored within the IBM Cloud Object Storage for consumption by the Algorithm Bench Node.js application.

When the consumer requests a trade, information about the user’s league, roster, and competition is used to personalize the broad player valuations. We use the concept of cross penalties based on a trading partner. The roster of the trading partner is analyzed for the number of players already available at a certain position. The importance of a position is reduced with the more players at that position on the roster. Through experimentation, three weighting groups were created around different positions.

The final personalized player valuation includes boosters and penalties based on league rules, roster composition, and the competition. The personalized valuations are used in an optimization bin packing algorithm to generate trades.

Player costs

Each trade comes with an opportunity cost. Teams in a trade gain value through received players but also send players away to an opponent. We measure the cost that a team will incur if the trade is accepted.

To start the process of determining the cost of each player on a roster, we determine the importance of each position given a roster.

Next, we need to determine the relative scoring contribution each player is projected to provide to the overall team score. Similarly, we determine the overall score contribution a player will make given the entire team.

The cost for a player is determined by averaging the cost metrics. We also personalize the costs based on a player’s roster, league rules and competition.

Trade partner pairings

Optionally, Trade Assistant with Watson enables the ability to pair teams together based on dissimilarity. Each paired team has the opportunity to offset each other’s weaknesses by accepting Watson trade recommendations. For each player on a roster, a positional-based value vector is created. The vectors include statistics about the players and team.

Each of the valuation vectors are input to determine the strength of a position on a team. Each of the strengths are calculated from a machine learning model.

Now that we know the strengths, we need to determine the weaknesses of the teams in terms of positional cost. Each of the positions is summarized based on the current roster makeup of a team.

A team’s strength and weakness at a certain position is concatenated together across all positions. A single vector describes the strengths and weaknesses of every team within a league, icon. A similarity metric is used to find teams that are the most dissimilar and likely to offset each other’s strengths and weaknesses. We chose to use the cosine similarity measure.

Math equation

Each of the pairwise teams are sorted in descended order from 90 degrees. The top paired teams are used together as trading partners. The number of trading partners is determined based on the practical amount of time a user can wait for a trade suggestion.

Trade packages

Now, trade packages can be generated after player cost, valuation, and teams are paired together for trading. A classic optimization bin packing optimization algorithm is used to optimize the trade value while minimizing the trade cost subject to the maximal cost a player is willing to risk.

Formally, we run an algorithm to get proposed players for a trade from the opposing team by maximizing total valuation. For example, we might have items with a cost and value. The objective is to maximize each item’s value while minimizing a cost. In classic problem formulations, bounds are maintained on cost.

The trade package generation algorithm is run twice and focused on each team within the pairing. The resulting players for both sides of the trade are constructed within a trade package. Each of the trade packages is measured in terms of fairness and value. The fairness metric provides a selfish view of the value you receive regardless of your opponent’s point of view. Fairness is a metric that considers both the benefit and realism for both sides of a proposed trade. The estimations help users make a trade-off between “what is in it for me” as contrasted to “will this trade be accepted.”

Next, let’s learn how we trained Watson.

FEAT and objective measures

The landing page for the Football Error Analysis Tool (FEAT) Figure 3. The landing page for the Football Error Analysis Tool

The generation of trade packages has many objective trade-offs that must be balanced by subjective human interpretations. The Football Error Analysis Tool (FEAT), as seen in Figure 3, was used in five error analysis sessions to gather feedback about generated trade packages. Over the sessions, we had at minimum five ESPN and five IBM evaluators that rated each set of trade packages that were assigned to them. When rating the packages, each review considered:

  1. Parity or fairness trade evaluation: A comparison between a team’s trading asset within a trade package.
  2. Likelihood of trade: The probability that a trade will be accepted by the managers of both teams through a deep learning model.
  3. Pain index: A measurement that determines how painful an overall trade is to an owner with respect to player cost and valuations.
  4. Value index: An overall egocentric view of the returned value in a trade from both a deep learning model and player valuation difference.
  5. Risk capacity: The risk that an owner is willing to take. For example, we define risk as the maximum amount of cost an owner will assume.

Other factors that were considered while rating trades include player performance forecasting, roster composition, injuries, upcoming games, next opponent rank, current position, perceived player brand, and league type.

When evaluating history information, each analyst had to minimize bias about players because we already know the outcome. For example, a player in the 2019 season during week 1 might have a very high rating but was injured in week 2 and out for the season. While evaluating week 1 trades, the known injury to a player in week 2 had to be removed from consideration.

After a user selected a week, league, and team, roster information and generated trade packages are available for viewing. On the roster view, a team’s detailed player inventory can be reviewed to get an assessment of each football player’s valuation, cost, and importance. With the roster context and as shown in Figure 4, the trade package view displays up to three trade packages for differing risk levels associated with a single trade partner. Players that will be sent and received are displayed along with general statistics about each player. The valuation and cost of player movement provides a descriptive analysis of the trade.

A FEAT trade package recommendations view Figure 4. A FEAT trade package recommendations view

Each analyst rates each of the trades to help teach the system to discern between good and bad trades. Each evaluation includes a rating between [1,10] from the perspective of both teams. Optionally, a free text field enables each evaluator to input any comments on general patterns of trades.

Over several months, the overall trade scores hovered around 6 and 7 where 10 is the best. We eliminated the majority of tail of ratings between [1,3] by applying 14 filters based on analytical rules and patterns. Figure 5 shows the score distribution before the application of the post processing.

Overall trade ratings in FEAT where 1 is the worst and 10 is the best Figure 5. Overall trade ratings in FEAT where 1 is the worst and 10 is the best

Now that we have high-quality trades, we need to deliver them to the consumer with speed.

Consumer-driven demand

Fantasy football users can access trade packages on demand. With each new user request, a new trade package is generated that contains one trade package between each team in the league. Within each package, we can have multiple trade recommendations. Only the best trade recommendations that survive provide a diversity of fairness, and value trade options are shown to the user. To keep the user’s interest, the algorithm must run under five seconds with a load of 1,000 requests per second. The algorithm runs a deep search and attempts to pair all teams to a target team for trade package generation. If the pairing logic exceeds five seconds, the number of team pairings is lowered.

The trade-off between trade recommendation volume and quality maps to the risk a team owner is willing to take. In general, we accept a medium risk and apply filters to keep the higher quality trades. To achieve and support the runtime requirements, the algorithm was deployed to three Red Hat OpenShift Kubernetes Service sites with a total of 525 PODs. Each site has 10 worker nodes with 16 cores and 32G of memory. With six ingress nodes available per cluster at four cores and 16G, the trade assistant algorithm returns deep trade results in less than a second. Traffic is distributed to each of the sites with Global Traffic Management to reduce network traffic hops.

The response time of each user request is approximately 200 milliseconds. Figure 6 shows an average latency of 200 milliseconds. In addition, most of the trade requests return three to four trades.

Trade Assistant with Watson latency and average number of trades Figure 6. Trade Assistant with Watson latency and average number of trades

Start trading

Let’s make a trade!

Engage your colleagues and friends with trade proposals from Watson that consider many factors while providing you a spread of fair and value-based trades. At best, you can improve your team and build trust as a fair trader. At worst, you might become a subjective #shadytrader. Enjoy your thought provoking on-demand trades. See you in the championship round!