There are a couple of ways to look at this question—- ML is predominantly a predictive analytics methodology— generally speaking, you can “optimize” a process or a system’s performance with better predictive accuracy using ML— we do this at WMT by predicting, based on Supply Chain activity history, more accurately truck loads, miles, cases per trailer, containers, etc. we better position transportation capacity to reduce costs— that optimizes the transportation system. Specifically speaking, the “field of optimization” most typically implies “mathematical optimization”— i.e., optimizing an objective function subject to a set of constraints, e.g., linear programming and its siblings (nonlinear, quadratic, integer, mixed-integer programming). There are a plethora of well known, tried and true algorithms and modeling approaches for solving these kinds of problems. IMO, where AI and optimization intersect is when AI addresses “complex human decision-making” then optimization models and algorithms can be embedded in broader, larger scope and scale AI solutions— I address this in the book with regard to AI augmentation (vs automation) where a human and a model work together iteratively, interactively to solve a complex problem— interactive optimization is a field, pioneered at Georgia Tech in the 1980s) that uses optimization-based models and heuristic algorithms that run very fast and achieve “good solutions quickly” that a human can evaluate, perform exception handling on, adjust some input data parameters and rerun the model. AI then creates new opportunities to apply optimization-based models and solutions.

Since you can win the book at the end of the week, I would not buy it in advance. This is more like an appetizer…

Good question Wali, there are many, many factors, well over 100. What we tried to do was distill it down to the main ones (following like an 80/20 rule) and the main reasons are the ones that you’d expect:

1. Not solving a problem aligned with the strategy
   
2. Not having the needed data
   
3. Not having the right people
   
4. Not having the right tools

So what is surprising is how often this still occurs in spite of all the frameworks, knowledge and research

Also see Doug’s comment below

Great question Ahmed, to add to what Doug wrote, I would say develop skills beyond just the technical. This is a big message in our book. You’ll need to learn how to do project management, stakeholder management, excellent communication, learn how to work with ambiguous problem statements.
The most important mindset is knowing that problems will be handed to you in a half thought out, vague and ambiguous fashion, learning to deal with things like that is a great start.
You can use a different mindset for future success too by simply setting expectations in line with what I wrote above. If what happens is what you are expecting, then it won’t be as painful.

Agree! Your emphasis on dealing with ambiguity and honing communication skills is spot on. At my work, I’ve encountered situations where project goals weren’t fully fleshed out, and it was a challenge to align everyone!

As Doug wrote, for us personally, luckily not. We’ve both been fortunate to work for analytically mature organisations

So you worked in both to archive the 80%?

I have worked in analytically mature companies like American Airlines and Walmart, and companies that were less mature, so yes.

But many examples in the book come from colleagues stories that run the gamut from large Fortune 500 companies to start ups that also range from analytically immature to mature.

I love the idea that adopting a “fail fast and learn” mindset can foster innovation!

Hi Aizzaac, for this one it is a question of setting expectations. With a given set of resources, it takes only so long to build something so people need to be aware of what is possible with the given constraints. The solution is likely to stage gate it in an agile fashion and release functional code along the journey that does a small, high quality solution for a piece of the problem. I really feel this is typically a stakeholder engagement/expectation type of problem when stated this way. Do you have an example of this for yourself?

No, I always run out of time. 😅

There is an old saw in software engineering that also applies to DS projects… “Good. Fast. Cheap. Pick two.” Agile MVP (Minimum Viable Product) is the best approach for building and delivering models, solutions rapidly with minimal time to value. We talk about Minimum Time to Value (identification, validation, then capture). Once value is identified that can buy time and garner support for investment needed to productionalize the solution. That said, there is no substitute for rigorous testing (unit, integration, UAT), MLOPS, pipeline development and stabilization. My quote from the book is relevant here. “Building models is easy. Building models into enterprise production systems is difficult by 10-100X” in cost, resources, complexity. MIT recently produced a research report that validated that 1-2 order of magnitude multiplier to get to production.

This will come from the company, you should be implementing the ethical guidelines set by the organisation. It is not hard to take a code of conduct or similar ethical policy and ensure that, when designing your system, that it meets these requirements. This should occur first and be part of the scoping process that should then set the timelines and deliverables before they are set in stone.
Do you have any specific examples you are thinking of in your own work?

We were tasked with developing a predictive model to identify potential fraud in financial transactions.
So, we scheduled specific phases for bias detection and mitigation techniques, such as fairness testing and algorithmic audits.
But as usual,we ran out of time 😅

The ones that are doing it already are things that automate processes like AutoML, (manage, what used to be, manual processes) like database versioning e.g. DBT and these are having an impact now and will have more in the future as companies adopt them. But especially generative AI tools that are able to generate quality code will help speed up development
Any specific use cases that you are thinking of?

Automated Explanation Generation: AI can generate human-readable explanations for complex models, improving transparency and trust.

Interesting it sounds like a combination of generative AI and something like LIME or SHAP, I haven’t heard of it but it reads like a ChatGPT created kind of concept

Well, some already stablished companies (Recorded Future, Spark Beyond) just connect a ChatGPT on the output of their models and showcase some “explainability”.

H2O.ai for example is an advanced AutoML platform that generates detailed reports in prose/English that explain predictive model results.

I wouldn’t say this is a data science specific question, this is a generic problem across all disciplines. One example that I have seen is that making people aware of biases can assist in reducing their effect.

There are field guides that walk you through how to diagnose a problem and figure out which model to use— here is one from Booz Allen https://www.boozallen.com/s/insight/publication/field-guide-to-data-science.html

I have built some of my own tools that look like a Pachinko Machine— you drop the coin (problem) in the top and it filters down into a bucket at the bottom that suggests appropriate model forms, tools, algorithms— using criteria such as Predictive vs Prescriptive, Stochastic vs, Deterministic vs. Hybrid

We built a tool at WMT that uses ChatGPT to recommend LLM model forms in response to a problem description prompt.

In the optimization world (prescriptive), Garey & Johnson’s book Computers and Intractability: A Guide to the Theory of NP-Completeness is the industry standard for diagnosing super-complex problems and mapping them onto well known problems, e.g., airline crew scheduling maps to set covering/partitioning and is usually solved with an integer programming formulation using column generation.

This is a skill, and an art form, as much as it is science, that must be honed and developed over time by working with people more experienced, more skilled than you are— either professionals, or academics (that understand and have worked on real world problems). The good news is that today for predictive problems, if you have clean data, AutoML tools will generate the best model (or ensemble of models) for you with the best independent predictor variables to achieve the greatest predictive efficacy— you still need to understand how to interpret and apply the results but it does cut down on time and errors in coding to get to a useful model form.

Good question, the answer comes back to the business questions. Just because there are new technologies and techniques does not mean you need to learn them or use them if they are not relevant (especially after doing a ROI calculation on how much effort they are to learn, their cost to use, etc that can quickly filter out uncommercial options). Narrow down to those that are relevant and then just understand them. The next step is what you would expect - trial them, test them in PoCs, etc.

I outline a methodology in the book on how to estimate value before the project begins. Having a BIG multiplier value target helps a lot to start with, like a big expense category— like jet fuel spend for a $20B airline is $6B— you establish a maximum potential benefit of ~ 1-10% and then take a % of that as your goal to reduce cost…

The churn - pricing change example is a great one. It is very difficult to get to as the economists say ceteris paribus (all other things equal) experiment conditions in the real world. I think you estimate churn for a given price structure, then rerun the churn experiment with the new price structure. I think an experiment to understand price elasticity of demand can be factored into churn. Experimental design is critically important and I address that concept in the book with two real world examples. Addressing confounding factors and properly “blocking” experiments, and using concepts like double-blind studies, control groups, etc. are critical to ensuring the integrity of outcomes.

The data owner(s) include and range from the data steward (usually a business person/SME/data domain expert), to the data governance person responsible for metadata, data lineage, data transformations, etc., the VP of Data Engineering responsible for data infrastructure integrity, the CISO responsible for data security/access controls, and last but not least the CDO—Chief Data Officer. The Data Scientist plays a role in maintaining data integrity, certainly on what happens to the model outputs, but they are in effect a customer of the data inputs.

I recommend Agile Kanban for DS modeling projects, and Agile Scrum for system development projects. I have found “planning poker” (using Fibonnacci numbers 1,2,3,5,8,13,21) to be a useful approach for assessing complexity at a task level and estimating level of effort for scrum planning purposes. The downfall of Waterfall largely for software projects in favor of Agile/SAFe is the answer to your question— breaking down large, complex components and tasks into smaller, more manageable chunks is a key— lots of data on team progress and performance (burn down charts, etc) against tasks and milestones on the timeline, lots of unvarnished, open, honest communication and feedback about how things are “really going” and where adjustments/course corrections need to be made— short, iterative dev/test cycles, like 2 week sprints, where you get feedback and can make course corrections before things get too far out of control are all critical factors to project success. Estimates about the unknown are always wrong— it is really a question of how far off is the estimate from reality, and the sooner you can figure out how far off you are and then course correct 1) take a delay and move the deadline out 2) assign more resources and break up the big rocks 3) refactor the code… an old rule of thumb from Fred Brooks in The Mythical Man-Month was that you should spend 1/3 of your time doing the design, 1/6 of your time coding, and 1/2 of your time testing (unit, integration, UAT, verification, validation, post-deployment tests) STILL holds true today IMHO. Also knowing your people and their skill levels— in absolute and relative terms across levels (Senior, Staff, Principal) and even within levels— not all engineers are created equal— hence the term Mythical Man-Month… there is no such thing as a “man-month” rather how long will John take to get this done vs. Judy or Jamaal.😀😇

Doug Gray thank you very much

Hi Daniel, sorry for the non-threading, I’ve sent Doug an email letting him know how.

Alright, thanks!

Interesting point and insightful use case, thank you Doug Gray and Evan Shellshear!

There are so many, I heard a hilarious one the other day when one of Australia’s AI leaders got up and told everyone how Shell was using Generative AI to optimise wind farm design. This showed a clear lack of understanding of the different technologies (completely unrealistic use of the technology)

I wouldn’t say that practitioners need a deep understanding, I don’t think we claim that anywhere. The most important part is realistic expectations which might require some high level knowledge

That’s a complex one and one that we address in the book

Via a business case, also see Doug’s good responses to these questions below.

That’s excellent and a core issue we raise in the book and Doug’s response below nails this one.

Hi Aizzaac, I think Doug partially answered this question above when he wrote:
https://datatalks-club.slack.com/archives/C01H403LKG8/p1732035785245949
But I think the way to do it is the allocation of some budget for innovation. This is not just a data science problem but one that is across any new risky project in the business that is more than just business as usual

Good question. I don’t think this is something that organisations just communicate and unfortunately when we talk about organisations communicating stuff like this usually we mean the leadership communicating which is normally the group to whom the complexities need to be communicated (or better yet understood).
It is an education piece and the data scientists themselves won’t be able to do this on their own (i.e. communicating upwards) and hopefully things like our book will be read by leaders and provide this education.

By ensuring we align them with the business, we talk about the greatest reason of failure in our book being around lack of alignment with a business problem, and metrics that don’t align with business priorities have a much lower chance of being actioned as they are deemed unimportant

My suggest is that you need to fix the data quality issue and apply metrics on that. Garbage in, garbage out as they say 😄

DS professionals to be most effective need to be intellectually curious about their industry, company, department, and problem domain. Understanding the business problem (or not) is the first reason DS Projects Fail. DS is a means to an end of delivering business value, not an end in itself building models and doing the math. Economic impact is the end game and in order to achieve that DS folks need to engage in understanding the business. Multiple good examples in the book on this front. DS is a support function to the business and needs to be embraced as such.

Gauging complexity is challenging— it goes back to deeply understanding the underlying business problem— then mapping that problem onto a well known, well understood problem— e.g., like Traveling Salesman Problem (TSP)/Vehicle Routing Problem (VRP), or set covering/partitioning for airline crew scheduling, or job scheduling on machines— complexity also stems from the number of integration points for data and other systems integration, dynamism, mission criticality— we cover this in the book in the section on moving from Sandbox to Production— rigor and thoroughness of analysis is the only answer I can give to properly ID complexity— timespan varies a lot depending on data availability and the above factors— my DS pros regularly build models in the cloud in hours, days, or a week, once the data is in place— a typical DS project end to end may take 3-6-9-12 months depending on data, modeling, testing, deployment, environmental constraints, resource availability— XXXL projects like UPS ORION (TSP/VRP) took 10 years and cost $250M to build and deploy, but saves $300-400M per year in costs. The best approach is to be conservative, realistic in your estimating process. If the value is there, the business should be prepared to stick with the plan, program. That requires persistence, commitment, and a champion.

Thank you for the reply, Doug! All valid points and I completely agree. The book sounds very interesting, will be sure to give it a read.

This is the challenge (and opportunity) with analytics, data science in general— when we “let the data speak”, inconvenient truths are revealed, long held beliefs and assumptions are invalidated, tried and true heuristics and rules of thumb no longer apply— the question is a cultural one— do you work for a company that is run by HiPPOs? Highest Paid Person’s Opinion— or is the company authentically fact-based, data-driven, model-oriented in the way it solves problems, answers questions, makes decisions? It is a HUGE distinction— but as DS we must be able to clearly, succinctly, diplomatically, rationally “speak truth to power” in English, in business terms they will understand, utilizing a variety of tools, like before and after (the model) experiments, visualization with charts, graphs, heat maps, etc. (this is HUGE, as a picture is worth 1,000 words), storytelling in the business problem/question/decision context— if stakeholders want to ignore the facts/data/model results, and hold onto their biases, you should probably look for another job somewhere else.

I agree! Establishing a culture of data-driven decision-making has been a key focus for us, though it can be challenging, especially when teams make fast critical decisions based on empirical evidence that later proves incorrect.
As you pointed out, visualizations and storytelling are invaluable in overcoming these challenges. I feel like data scientists play a vital role as champions of this culture, helping to promote and embed it across the organization, especially in companies where this mindset isn’t yet deeply ingrained and has gained focus recently mostly because of the current AI hype. Perhaps another reason why soft skills should never be ignored as well haha thank you for sharing your insights!