On March 28th, we have delivered the first release of Decision Optimization for Data Science Experience (DO for DSX). Data Science Experience Local is now the only platform to include all technologies required to support Predictive and Prescriptive Analytics.
Decision Optimization for Data Science Experience helps organizations prescribe better decisions by making it easier for data science teams to build, validate, and deploy optimization models.
For that DO for DSX relies on 4 major sets of functionality:
- Built on top of DSX Local
- Build Decision Optimization models
- Validate Decision Optimization models
- Deploy Decision Optimization models
This post is a quick overview of these 4 major sets. Further posts will go into more details on each of them.
DSX Local is a collaboration platform for Data Science teams. Clusters allow several data scientists to create and share data and models through projects. Smooth github integration makes it possible to easily share content externally.
DSX Local includes all the useful open source programming tools : Jupyter and Zeppelin notebooks, Python, Scala, Spark, R, RStudio, etc along with add-ons from IBM such as SPSS modeler flows and Data Refinery for data transformation.
In order to create a complete application, it is now possible to use Decision Optimization along with some other technologies from one single environment.
With Decision Optimization, unlike with Machine Learning, the models are not automatically trained, but they need to be formulated, in accordance to business rules and objectives.
The preferred language to formulate the optimization model is Python, which is a de-facto standard in Data Science. For that, docplex python package is pre-installed.
Models can be created in notebooks where data scientist can use all his preferred packages to transform data or display solutions.
Python models developed in notebooks can also be imported in the DO model builder where different scenarios can easily be created and displayed or compared using dashboards. The model builder provides four simple steps to import and prepare data, import and solve a model formulation and display the solution.
If you are completely new to optimization, we introduce a modeling assistant to help you formulate your model starting from your data mapped to some specific problem domains.
With Decision Optimization, unlike with Machine Learning, model validation is not automatic, based on some historical data which has been put aside of training. Of course, some historical scenarios can be run automatically to ensure the right solution is found. But, initially, some interaction with the model and different data is required in order to validate it.
For that, DO for DSX provides dashboards, where widgets can be organized using different pages and layouts to represent data or solution using tables, charts or Gantt chart.
Different scenarios can also be created, for example to compare how a model formulation behaves with different data, or the same data on different model formulations, with different constraints or objectives. The differences can be easily displayed in the dashboard.
In case you need more, for example to run a large scale simulation, a python API is provided so that it is simple to automate creation, modification and/or solve an important number of scenarios.
When model formulation has been validated, the next step is, in general, to deploy and integrate it into a production application.
DSX Local, with its new Model Management and Deployment (MMD) module, makes deployment of optimization models trivial. In a few clicks, a micro service is set up with dedicated entry point and resources.
Using github tags, the control of which precise version(s) of model to deploy is simple and precise.
Deployment Manager then allows administrator to create one or several release package(s) to deploy one or several version(s) of your model.
In all cases, administrator is in control of the resources dedicated to each model (cores and memory), and he can monitor the behavior of these deployed models.
For someone developing optimization models and deploying optimization based applications, there are two levels of improvements.
On the one hand, the development, validation and deployment of optimization models has become much easier, based on new modern technologies you can really quickly obtain a micro-service for your model.
But on the other hand, it has also become much easier to develop and deploy both predictive and optimization models from within the same environment, and deploy them according to the same architecture. This will be better illustrated using a specific example.