Optimize Decisions with ML: Prescriptive & Robust Optimization for Supply Chain and Pricing | Dan Becker

The transcripts are edited for clarity, sometimes with AI. If you notice any incorrect information, let us know.

Alexey: I’ll just do a short introduction. On the slide, it says "Decision Optimization" because the full title was too long to fit. The actual title is "Translating Machine Learning Predictions Into Better Real-World Results with Decision Optimization." This talk will be different from the three previous ones we had. (0.0)

Alexey: The other talks were presentations with a Q&A session afterward. This one will be a live discussion and will also be released as a podcast on DataTalks.Club. Unlike the previous videos, this will not have a video version on YouTube; it will be just a conversation. (27.0)

Alexey: For those who just joined, we use Slido for questions. On YouTube, in the chat section, there is a pinned message that links to Slido. At any time during the chat, you can submit a question there, and we will cover it eventually. (54.0)

Alexey: Are you ready to start? (1:19)

Alexey: Dan (1:19)

Alexey: Yes, let’s start. Today we’ll talk about decision optimization, which is about making better decisions with machine learning. We have a special guest today, Dan, the founder of Decision AI, a company that specializes in improving decisions using machine learning. (1:19)

Alexey: Welcome, Dan. Glad to have you here. (1:47)

Alexey: Dan (1:47)

Alexey: Thanks for having me. (1:47)

Alexey: Before we dive into decision optimization, can you tell us a bit about your career journey so far? (1:53)

Dan: I got started with machine learning in 2000 at a startup that used it to help companies optimize their listings on eBay, which was very popular at the time. We worked on machine learning for about six months, but it was a complete failure. The predictions were not good, and no one really understood what we were doing. After six months, we switched to using simple descriptive statistics. (2:04)

Dan: After about nine months, I thought machine learning would never catch on. It sounded cool, but I didn’t think it would work. I then pursued a PhD in econometrics. In 2009, I started competing in Kaggle competitions, initially using simple statistical techniques because I thought they were best. I quickly realized I was in almost last place, and machine learning had improved dramatically. (2:33)

Dan: I eventually got second place in a competition with a three million dollar grand prize, but no prize for second place. That experience led me to consulting with several large companies. I was an early employee at DataRobot, worked at Google as a data scientist, and in January 2020, I started Decision AI. (3:07)

Alexey: You started Kaggle in 2009? (3:38)

Alexey: Dan (3:38)

Alexey: Yes, it was probably one of the first competitions. At the time, there were maybe three million users, and my user ID was 9000 or 9028. I did well in the early competitions, but now the competitions are much more complex and competitive. (3:38)

Alexey: I started Kaggle in 2015, six years after it launched. Looking at competitions now, they are very different from what existed five years ago. The level of complexity has increased significantly. (4:14)

Alexey: What is decision optimization, and what kind of problems does it solve? (4:34)

Alexey: Dan (4:34)

Alexey: Decision optimization is essentially what it sounds like. One way to explain it is through an example. A few years ago, I spoke with a friend at a large U.S. airline. Their most important problem was setting prices for flights. (4:34)

Dan: They built machine learning models to predict how many tickets would sell at different prices. For instance, Flight 1000 leaves on March 20th. If the price is $400, how many tickets will sell before the next price reset? If it’s $500, how many will sell? (5:11)

Dan: Decision optimization takes this a step further. You have a machine learning model that predicts sales, something you control like price, and an objective like total revenue. The challenge is choosing the price to maximize that objective, considering real-world dynamics over time. (5:51)

Dan: For example, if you have 90 days before the flight, tickets not sold today can be sold tomorrow. If you lower your price too much today, competitors may lower theirs, reducing future revenue. Decision optimization answers which action maximizes overall performance. (6:27)

Alexey: So, you are saying we have a model, like a demand forecasting model, that predicts seat sales for a given price. The problem is how to use this model most effectively, avoiding losses from selling tickets too cheaply or too quickly. (7:01)

Alexey: Does this also work for other types of problems? As a data scientist, I mostly deal with binary classification problems, which I think are common in industry. (7:49)

Dan: Yes, let me give two examples. For a binary classification problem like financial fraud, a company provides an API that predicts the probability a transaction is fraudulent. (8:24)

Dan: The question becomes: what do I do with a transaction that is five percent likely to be fraud? Accept, reject, or investigate further? In the simplest case, decision optimization helps set the threshold for rejecting or accepting transactions. (8:58)

Dan: This approach can significantly impact business outcomes. For example, an insurance company uses a threshold to decide which claims to investigate. Even a simple threshold-based system can be very effective. (9:46)

Dan: However, in practice, the situation is more complex. If a claim comes in for 50,000 euros and is nine percent likely to be fraud, they might not investigate because it is under the 10 percent threshold. Meanwhile, smaller claims may be treated differently. (10:33)

Dan: It is 10 percent likely to be fraud, so the expected value of investigating the nine percent is about 98 to 90 times as large. Yet they don’t do it because we focus too narrowly on classification output as a probability. I see people make mistakes because they look at that probability in isolation. (11:01)

Alexey: Shouldn’t the model be able to account for the potential value, like considering the sum or overall impact? (11:39)

Dan: The probability reflects the likelihood of fraud, but the value of investigating is different. For example, if a claim is 10 percent likely to be fraud and the amount is 500 euros, the expected value of investigating is 50 euros. So, even a small probability can have a high expected value depending on the amount at risk. (11:45)

Dan: We need to do more than just look at probability. At the company where I work, we have a moderation team for an online classifieds platform. When someone publishes a listing, we want to determine if it is fraudulent. If it is, a moderator or automated system investigates. (12:24)

Dan: Typically, we set thresholds using precision and recall. We choose a level of precision and recall that is acceptable and then roll out the system. However, this is not always ideal. (13:00)

Alexey: How should we approach this differently with decision optimization? (13:32)

Dan: Decision optimization is very use case specific. Not every type of fraud has the same impact on a platform. You need to assign a value or cost to each type of fraud. (13:37)

Dan: For example, one post might be a 10,000 euro racing bike, and another a low-cost cruiser. Even if the probability of fraud is the same, the potential loss is different. You might prioritize protecting someone from losing the high-value item. (14:03)

Dan: Similarly, if the risk is physical harm, you need stricter standards to protect users. The point is to assign values to different outcomes. Once you do that, you can incorporate those values into your decision-making. (15:03)

Dan: So we attach a value to each item or post. Next, we combine that with predictions from our classifier. We now have two inputs: predicted probability and value. The goal is to create a single decision function that takes these factors into account. (15:27)

Dan: Think of it like a Python function. It takes inputs like the probability of fraud, type of fraud, and item value. You may also factor in customer importance or repeated behavior. The output is accept, reject, or manually investigate. (16:00)

Dan: This process varies by company. At tech companies, banks, and insurance companies, the people making decisions differ, but the idea is the same. You simulate what happens under different decision rules. (16:51)

Dan: We propose several decision rules and see which outcome is most desirable. You may also consider metrics like lifetime value or the fraction of users who leave if upset. Data scientists often make the mistake of treating each model in isolation. (17:30)

Dan: For example, one model predicts likelihood of fraud, another predicts user reaction to a mistake, and a third predicts lifetime value. You combine these models to evaluate overall outcomes. Each piece interacts to reflect the real-world system. (18:15)

Dan: At the end of the day, you want a single decision function, like a Python function, to simulate outcomes and see which one is best. You can use it for actual decisions or experimentation. (18:52)

Dan: In some businesses, A/B testing is possible and reliable. If not, you can simulate outcomes in a sandbox environment. The simulator allows you to try different decision functions and see their results. (19:19)

Dan: The decision function is what will eventually be deployed. You can play with its parameters and observe outcomes. This is the core of what Decision AI specializes in. (20:09)

Alexey: Who should create this decision function? Should a data scientist define it? Isn’t the point of machine learning to avoid hand-crafting rules? (20:41)

Alexey: If you were to build a machine learning model, what is the API for it? Whether you use XGBoost, TensorFlow, or scikit-learn, one thing is consistent: it always has .predict. In plain English, that answers the question: what will happen? (21:06)

Dan: Compared to models from ten years ago, like linear regression or GLMs, we used to do a lot of manual feature engineering to improve predictions. Machine learning automates the prediction of what will happen. Now, the question becomes: what should I do about it? This is very different from predicting outcomes. (21:25)

Dan: Reinforcement learning is a branch of machine learning that optimizes an objective in a complex environment. Someday, reinforcement learning might solve this for us, but we are not there yet. Most breakthroughs in reinforcement learning, like AlphaGo or OpenAI’s Dota agent, are in games because games provide a simulator. (21:58)

Dan: Simulators let you try different rules in a dynamic environment and see how things progress over time. We need similar simulators for dynamic real-world environments. You cannot take a conventional supervised model and expect it to optimize for broader objectives. Supervised models are too narrow for this purpose. (23:03)

Alexey: So ideally, we would combine our supervised models into one decision function and train it with reinforcement learning, but we don’t yet have an environment to experiment properly. (24:01)

Dan: Exactly. Right now, we usually do this manually, encoding rules in a decision function. Some rules are straightforward to humans but hard for machine learning to learn. The best approach often combines known rules with machine learning predictions. (24:27)

Dan: For example, in the airline case, total flight revenue is the sum of revenue today, tomorrow, and the next day until departure. This is simple to encode manually, but a machine learning model would struggle without the right data. In many cases, domain knowledge is more efficient than learning everything from scratch. (25:00)

Dan: The person providing the knowledge is often a domain expert, not a data scientist. The process varies by company. In tech companies, a data scientist may collaborate with a product manager. In other businesses, it could involve multiple stakeholders. (26:13)

Alexey: What tools are available for this? Does your company develop a tool for decision optimization? (27:18)

Dan: Yes. The primitives are your machine learning models. Then we have a simulator that incorporates these models to predict business-level outcomes over time. The last part is the decision function. You take your models and drag them into our web-based app. (27:51)

Dan: We have a domain-specific language to write formulas. For example, the number of tickets to sell today is whatever we had yesterday minus what we sold. You use .predict from the demand model as input to these formulas. (28:34)

Dan: The simulator propagates information over time. It calculates outcomes for day one, then simulates day two based on competitor prices and previous sales. It continues sequentially, propagating predictions through all days. (29:26)

Dan: The exact behavior depends on the formulas you wrote. You could implement this in raw Python, but iterating is difficult. You need to calculate in the correct order, piping information from one calculation to the next like a DAG. Maintaining it manually is very complex and messy. (30:02)

Dan: Solving this iteration and maintenance problem is the main challenge we addressed with our tool. (31:53)

Alexey: With your domain-specific language, how are companies using this now? Are there open-source tools, or do people just write scripts in Python to create these environments? (31:58)

Dan: There aren’t great tools for this, which is why I started Decision AI. At Google and a few other companies, people either use raw Python with NumPy for calculations and connect it to TensorFlow or PyTorch models at a low level. I’ve also seen proof-of-concept approaches using probabilistic programming languages like PyMC3 or Pyro. (32:16)

Dan: So basically, the alternatives are to encode everything manually, or in large companies, maybe they have internal tools. This is not widely available for general use. Our software, by the way, is available for free, and it allows iteration 10 to 30 times faster than raw Python. (33:16)

Alexey: Coming back to your career journey: you competed on Kaggle, worked at DataRobot, then Google, and now started Decision AI. What did you see in the industry that led you to start your company? (34:00)

Dan: I noticed that in 80 percent of applications, there is an important decision step after the machine learning model that is often overlooked. Many data scientists deploy models, but the business may not use the results effectively. For example, at the airline my friend worked for, his group was cut because they could not turn accurate predictions into better business outcomes. (34:37)

Dan: Twenty percent of the time, turning predictions into decisions is almost obvious. But 80 percent of the time, there is a huge opportunity. Machine learning has improved predictions significantly, but how you use those predictions is equally important. Five years ago, when I suggested simulation approaches we used at Google, people found it too manual. (35:30)

Dan: Custom tooling didn’t exist, so I realized there was a strong need to help people make better decisions with models. Researchers focus on small improvements in accuracy, like 0.91 AUC instead of 0.9, but in practice, the value is often wasted if the model is not used effectively. (36:19)

Alexey: That resonates with what I see. How can a data scientist start integrating this into their workflow? (37:13)

Dan: The first step is to start small. For a binary classification problem, talk to someone in the business to assign a monetary value to every false positive and true positive. You can transform your confusion matrix to see the impact of different thresholds. (37:57)

Dan: For each threshold, calculate the expected monetary value. You don’t need our software for this—it’s the first step. Even thinking rigorously about optimizing the decision threshold, the same way you are analytical in modeling, is very valuable. (39:00)

Dan: If there are dynamics over time, like customers reacting differently tomorrow based on today’s decisions, a simple threshold isn’t optimal. Being rigorous about a single threshold is better than nothing, but letting it vary with the environment is the next step. (39:41)

Dan: I recommend checking out our software. You can use it for free. Even the mindset shift of optimizing how you use your model for decisions is valuable. It lets you model impacts on profit and loss or customer metrics a year from now. (40:12)

Dan: Even imperfect modeling is a big step forward. Our website is www.decision.ai (40:52)

Dan: , and our app is app.decision.ai. I recommend exploring it to approximate the ideal way to make decisions based on your models. (40:52)

Alexey: I actually meant slightly different thing (41:23)

Alexey: Let's say we've played with Decision AI and found a reasonably good decision function. Do we export it and use it in production, or how does it look? (41:29)

Alexey: Dan (41:29)

Alexey: Yes, you deploy it as a RESTful endpoint. It takes the inputs for your decision function and returns the output, like “accept this transaction” or “reject it.” From the rest of the organization’s perspective, it looks like a standard web service or REST API. Features go in, decisions come out. (41:29)

Alexey: Who usually drives this initiative? Should it come from a data scientist or a product manager? How can a data scientist explain the value of this? (42:24)

Alexey: Dan (42:24)

Alexey: I think the initiative should come from the data scientist. They spend a lot of time building models, and if those models don’t pay off in business terms, both the data scientists and product managers are in trouble. Increasingly, companies expect data scientists to prove that their models drive profitability, not just improve RMSE or accuracy. (42:24)

Dan: It’s an analytical process. The data scientist should lead and bring in the product manager. Start by asking what the product manager cares about—daily active users, revenue, or some other metric. Then map predictions from your model to that metric. For example, certain types of fraud may hurt reputation, which affects daily active users. Once you show that you can optimize for the metric the business cares about, it’s usually an easy sell. (43:54)

Alexey: So it’s about metrics the business cares about, not just accuracy? (45:05)

Alexey: Dan (45:05)

Alexey: Exactly. Accuracy is only a tool. Start with what the business cares about, and ensure the model helps achieve those goals. (45:05)

Alexey: Are there cases where we don’t need decision optimization? (46:01)

Alexey: Dan (46:01)

Alexey: Yes. Sometimes predictions are “good enough.” For example, recommendation systems like Netflix or YouTube: predicting user engagement over the next six months can be accurate enough that a separate decision function isn’t needed. (46:01)

Dan: However, these are often very complex models, like YouTube-scale recommendation engines. In such cases, the decision logic is already largely encoded in the model. (48:02)

Alexey: What trends do you see in data science as a whole? (49:00)

Alexey: Dan (49:00)

Alexey: I work a lot with tabular data, especially supply chain problems. A major concern is train-test drift or concept drift. Historical data may no longer represent current conditions, especially after events like 2020. Models trained on older data can fail to predict accurately, so businesses must adjust quickly. (49:00)

Dan: Common approaches include model monitoring, where you track performance or covariate shift daily. If the current data differs from the training data, the model must be refreshed quickly. Another approach is programmatic adjustments based on hypotheses about how the world is changing. Explainability techniques, like SHAP values, can help make these adjustments. (50:37)

Dan: The world changes quickly, so it’s crucial to be thoughtful about relying on historical data. This is a complex problem for both data scientists and businesses. (53:59)

Alexey: How can people find you? (54:13)

Alexey: Dan (54:13)

Alexey: On Twitter, I’m @dansbecker, and I’m on LinkedIn as the founder of Decision AI. If you have a data science challenge, feel free to reach out—I enjoy discussing how to use machine learning effectively. Email me at dan@decision.ai (54:13)

Alexey: . (54:13)

Alexey: Thanks for joining us. This was a long session, but very informative. For those who stayed until the end, thanks a lot. For those who didn’t, the video will be uploaded separately. See you next week. (55:03)