Table of Contents

1. Preface
2. Key Insights
3. Conclusion
4. References
5. Appendix
6. Contributors

Preface

As machine learning (ML) is making a profound impact on the entire fabric of society and economy, companies in every industry are actively upgrading their technology stacks and business strategies to use insights from high quality data to provide a roadmap to stay competitive. Along with the ubiquitous adoption of machine learning in every industry, a new set of challenges have arisen when companies integrate machine learning into their business models. The alignment of business goals, technology, talent accessibility, data security and governance and affordability requires a delicate balance when adopting machine learning in business. A further continuum of a nascent subset of ML is also taking shape. RealTimeML, within a workflow is exploding with the advent of online predictive analytics as campaign engineers grapple how to commandeer these new technologies. We will introduce RealTimeML as a component of the ML Lifecycle later in this report and begin to monitor RealTimeML among our respondents beginning in Q2 of 2022.

Our ML readiness survey and scoring methodology are designed by professional machine learning experts, including the Loxz Digital data science team that has over 30 years of ML experience collectively. Specifically, our digital assessment questions speak to the entire machine learning lifecycle by dividing survey questions into six participating sections that represent different ML lifecycle components. By taking our ML readiness survey, you are able to get an immediate and comprehensive ML readiness score (MLR) and sub-score of each stage in the ML lifecycle, including data preparedness (DP), model development (MD), deployment models (DM), model monitoring (MM) and business value (BV) (See Table 1). As a result, we’ve further classified the respondents into one of the four industry roles regarding ML readiness; they include observer, performer, innovator, or leader. The overall assessment of ML readiness along with our innovative granular diagnostic sub-scores provides a comprehensive vision towards your organization’s ability to thrive with ML. Not only identifying your capabilities and requirements, but those of your industry peers.

Our survey has also attracted the attention of academic institutions, such as the Stanford Digital Economy Lab and UC Irvine, since the survey provides a good benchmark for academic institutions, to profoundly update their curriculum by hosting the survey and understanding from the data that is piped in, what areas of study within ML should these academic institutions should focus on. By 2025, 12M new data scientists will take new roles at organizations, who must be ML ready. These roles include but are not limited to Data Scientists, MLOPs, ML Engineers, Data Analysts and Data Science Engineers.

Loxz Digital’s Q4 survey report reveals the influence of machine learning in the enterprise from multiple respects and helps you forge a clear position to identify the opportunity and challenges regarding machine learning adoption. With insights from the survey results, you are able to better understand the strengths and weaknesses you have as an organization regarding the ML adoption process and improve it by leading more targeted efforts by finding the exact resources you need to thrive. We’ll dive deeper into these elements in this report.

Scoring & Sub-scoring Methodology

Our scoring methodology generates machine learning readiness (MLR) scores and sub-scores for constructs that can not only determine how your organization can bolster its MLR, but precisely identify your organization’s ability to leverage machine learning models in a quantifiable manner that ultimately generates business value. Beginning with Q2 of 2022 we will begin reporting on a new sub-scoring element. There will be several specific questions relating to RealTimeML in addition to the five current sub-scoring categories.

ML readiness score and subscores concept

Immediately, upon responding to our assessment, you are classified as an industry observer, performer, innovator, or leader. This categorization not only conveys the overall assessment of your organization's MLR but helps equip our recommendation engine with the information needed to make specific and more importantly timely recommendations your organization can follow to take your machine learning models to the next level.

The MLR sub-scores (See Table 2) can then provide targeted insight about how you can improve as well as a comprehensive assessment as to where you stand in the ML lifecycle. Taken together, these scores encompass comprehensive insight of not only your capabilities and needs but those of your industry competitors.

Developing the beta scoring algorithm

The Loxz Digital Organizational MLR survey covers a variety of specific questions which are factored into our scoring methodology, used for survey refinement, and ultimately to creative value for you as the respondent to provide your organization with important recommendations. With the help of domain experts, we identified thematic trends in organizational machine learning. These trends were then used to map constructs onto particular items. This process means that your organization’s scores are calculated based on a proprietary scoring algorithm vetted by internal industry experts and data scientists. By identifying the most important questions we can streamline the survey-taking process and help ensure that your machine learning readiness score accurately reflects your organization’s ability to successfully prepare, develop, train, deploy and monitor any machine learning project.

MLR scoring & sub-scoring Methodology

The Loxz Digital Organizational MLR survey is more than just a measure of your organization’s MLR, it's a diagnostic assessment that uses a tightly vetted answer key system to simultaneously increase accuracy while reducing bias. To establish an accurate score, our team of experts developed an answer code for every response for any designated scoring item. Each choice was hand-selected to represent the Machine Learning maturity of the organization within a particular dimension of MLR, such as an organization’s awareness of the importance of machine learning development and an organization's ability to deploy a machine learning model effectively.

The answer code is represented by values (ranging from 0-12), which are then transformed and weighted based on the relative complexity for having a resource of completing a task.

The lowest possible score for each item is always 0. The answer code increments by one, with a higher answer code indicating a higher level of maturity. This methodology is the same for each sub-score, allowing us to assess your organization’s machine learning maturity across the entire lifecycle. Weights established by our panel of experts are assigned to each question and used to calculate the final MLR score. We do this with the help of our domain experts, who assign a value to each question ranging from 1 (being the least important) to 10 (most important). The weights derived from our domain experts are then averaged and applied to the overall machine learning readiness score. The overarching method effectively reduces both the potential bias attributed to organizational knowledge silos where a select few determine the importance of items for scoring. This allows us to generate a score that is both reflective of the experience possessed by domain experts and fairly represents important nuances of an organization’s ability to carry out effective machine learning.

Composite construction. Categories that represent the ML lifecycle are assigned to questions by our domain experts allowing you to understand the pressure points and barriers that are inhibiting your MLR projects.

We partition all scoring questions into 5 subcategories using the same methodology employed to evaluate the maturity of machine learning readiness. Similar to the machine learning readiness score, each score ranges from 0 to 100.

Key Insights

Companies are actively innovating in machine learning but many companies take a wait-and-see attitude toward adopting machine learning.

As noted in Figure 1, more than 50% of the survey respondents are labeled as an innovator regarding ML readiness. The global machine learning market is projected to grow from $15.50 billion in 2021 to $152.24 billion in 2028, according to a report by Fortune Business Insights (Maruti Techlabs, Challenges faced by businesses in adopting machine learning).

The growing adoption of machine learning across industries is already bringing greater competition, which is one of the most important catalysts for innovation in machine learning. Even though adopting machine learning in business is a rising trend, some companies are taking a wait-and-see approach toward applying machine learning in product development because adopting machine learning means more than just developing machine learning models, but also possessing sufficient resources and successful strategies through the whole ML lifecycle. It involves plenty of elaborate planning and detailed execution. So some companies are careful with machine learning adoption, which has been reserved mainly for big tech companies.

Leaders generally outperform throughout the ML lifecycle but innovators are doing even better than leaders against model monitoring.

As noted in Figure 2, ML leaders score higher than other roles in all ML lifecycle stages except formodel monitoring.Innovators score slightly higher than leaders in the model monitoring stage, showing the fact that innovators put a lot of effort into enhancing the model monitoring process and had made some achievements. The achievements indicate that model monitoring has huge development potential and cultivation space for performers and leaders as well.

Model Monitoring is facilitated by specific tools in AWS, GCP and Azure, to not only monitor the performance of the model but also to identify thresholds in data drift or concept drift of the deployed models. There is an entire industry being borne on model monitoring and we believe this is the next stage of dramatic growth trajectory within the MLOps space. Companies that can excel in monitoring models, detect concept or data drift early and retrain models in realtime will have an edge.

The greatest reason that prevents observers from adopting ML in their business is that adopting ML is not the top priority at the current stage. Lack of resources and talent is an inescapable fact that observers are struggling to pursue ML.

As shown in Figure 3, 23.91% of the survey respondents indicated that they don’t consider adopting ML as a top priority and 39.31% of the survey respondents showed that they have struggled with a lack of resources, including financial resources, business resources, and specific talent pools to carry out ML adoption effectively. . Pursuing ML in the business environment requires not only technical support but also business support and while leadership among observers have not recognized the business value of ML yet, they will shortly, or risk being obsolete. ML touches every aspect of a company’s identity, from online predictions, to interactive demos of models trained, and must ensure that machine learning strategies and procedures align with long term business objectives. Business priorities and ML adoption within the organization could be influencing each other. ML development can be a positive force for business development, but at the same time, ML development requires specific talent and resources.. Thus, ML development, sometimes, conflicts with business priority among observers. Organizations categorized as “observers” must decide on the right timing to adopt ML as a priority. As noted in Figure 3, organizations are not ready for adopting ML because it potentially conflicts with the current business priorities.

Companies outshine in model monitoring stages regardless of the time to deploy an ML model.

Companies are beginning to prioritize model monitoring because model concept drift and data drift happens over time and it can largely impact a business and even destroy the user trust. Without model monitoring after deployment, companies cannot ascertain peak model performance or when thresholds of a model have been breached. As noted in Figure 4, no matter how long it will take companies to deploy an ML model, all the companies excel in the model monitoring stage. Notably, companies that usually spend 2-4 weeks to train and deploy a model get the average highest score in the model monitoring stage and are ranked second in the model development and model deployment stage. On the contrary, companies that take 1 week to deploy the model get the average highest score in the model deployment stage but fall into third place in the model monitoring stage. In the machine learning lifecycle, every stage is closely related to others, and the strategies for each stage should align with overall business objectives. As stated earlier in this report, we believe every stage of the ML Lifecylce will evolve as separate industries as more companies create platforms dedicated to model monitoring. You can find a growing number of sophisticated model monitoring tools in AWS, GCP and Azure as well.

Technical support along with business leadership support is vital in adopting ML in your businesses. Companies cannot reach a high performance whenever any one of these business segments are not aligned.

Companies with support from both deep MLOPs expertise and other technical teams get the highest score in all of the ML lifecycle stages except for the data preparation stage. The reasoning behind this is that technical support and leadership support play a larger role in model development, model deployment, and model monitoring compared to the data preparation stage. But as Figure 5 suggests, companies that have only business leadership team support score the highest in the data preparation stage. Building data collection mechanisms and articulating the problem statement at the early stage is critical in the data preparation stage, and both rely heavily on business decisions and support besides technical support. The result in Figure 5 shows a significant disparity between companies that only have a business team or only have a leadership team and companies that have all, indicating that cross-functional teams are one of the most important factors that determine the success in ML adoption.

Industrial innovators show an equal demand for interpretable machine learning models and complicated algorithms that are hard to explain. However, ML performers and leaders focus on interpretable models.

As shown in Figure 7, ~70% of theinnovators show that ML is interpretable to their products. But there is a noticeable gap in the number of companies among ML performers between identifying ML as an interpretable feature and identifying ML as a black box. Innovators seek more complex algorithms and tend to take more risk to meet the varied customer demands to develop and expand markets. On the contrary, leaders usually have stable business models and products, so they have little incentive to take risks to change the mode. Keep in mind that the growth trajectory among leaders with ML ( big tech) is quite different than those of innovators.

Machine learning is extremely important for companies that are in the E-commerce and retail industries. Similar impact is identified in most industries, including finance, healthcare, software, and education industries.

As shown in Figure 8, 100% of the surveyed companies that are from the e-commerce industry and retail industries show that they are solving problems using machine learning, meaning that ML is indispensable for their businesses. The results from the plot indicate that more than 60% of the companies surveyed, show that they are using ML in their business. However, in the media and energy industries, ML sees slower adoption as less than 40% of the companies that are from themedia and entertainment industries are using ML and around 50% of the companies that are from theenergy industry are using ML. The smaller-scale applications of ML, on the other hand, show potential in adopting ML in these industries. The world has witnessed the development of renewable energy in the past decade. With advancements in machine learning and AI, companies in the energy industry is able to predict weather conditions, energy consumption, etc. However, renewable energy development largely depends on the the distribution of energy resources and energy source itself, so even though machine learning and AI technologies are showing great potential in energy industry, limited scale of applications are seen.

Conclusion

According to the Digital Marketing Institute, 97% of business leaders believe the future of marketing consists of marketers and machines working in tandem to better a business's outcome(Fallon, N. 2020, May 12). As machine learning is playing a more important role in business, undoubtedly, companies are applying ML to boost business performance. The Loxz Digital ML readiness survey, along with the MLR score and subscores, provides you with an efficient and diagnostic assessment of your machine learning performance throughout the entire machine learning lifecycle. With a subscore in each ML lifecycle stage, you are able to gain insights into your company’s competitiveness and disadvantages and how to succeed in your ML adoption journey.

Throughout this report, we are providing you with an overview and understanding of ML adoption trends and roles of companies across industries, as well as key insights into opportunities and challenges that companies are facing. We see an obvious trend that companies are actively adopting machine learning in their business and innovating in machine learning. But the performance varies across industries and relates excessively to accessible resources, mainly talent. However, there are still a number of companies that are waiting for proper timing to adopt ML because of the priority conflicts and lack of resources and budgets. Adopting Machine Learning has a relatively high barrier to entry, if you consider the talent pool scarcity.

Another nascent component to Machine Learning is RealTimeML, which is a new trend in Machine Learning. At Loxz Digital, building real-time machine learning models is the bedrock of our business and serving online predictions within the workflow of a campaign engineer has enormous business value and fresh perspectives with benefits. We believe ML innovation is moving towards RealTimeML rapidly because companies outside of big-tech see the benefits of online predictions within a workflow. Streaming data pipelines and realtime inference within cloud environments are gaining popularity as these RealtimeML platforms are built and will replace traditional batch processing related ML transactions. There are a few challenges with RealTimeML online predictions, such as inference latency and MLOps tooling. We believe 2022 will be an important year for RealTimeML and the benefits to engineers who experiment with online predictions. No longer are these important RealTime insights relegated to big -tech.

Web 3.0 and ML

There is a great deal of smatter about Web 3.0. I wouldn't worry too much about it except to share that ML will be fixated into every facet of an organization's logic. From online predictions within a workflow of a campaign to RealtimeML recommendations. Companies who are becoming model building factories, will excel at a much greater pace as the logic layer of the technology stack becomes "smarter" and more adaptive, while the interface layer now mainly used for serving predictions becomes more natural and intuitive. ability to access RealTimeML models at scale and seamlessly weave or access these proven models together are potentially where the logic layer meets the network effect. Importantly, as we want to trust more than just the data records of things, we want to trust these algorithms as well. It all starts with the right data.

Take our ML readiness survey today and get your scores

References

Challenges faced by businesses in adopting machine learning. Maruti Techlabs. (2021, September 9). Retrieved December 27, 2021, from https://marutitech.com/challenges-machine-learning/

Fallon, N. (2020, May 12). How will machine learning impact your business? https://www.uschamber.com/co   Retrieved January 5, 2022, from https://www.uschamber.com/co/run/technology/machine-learning-impact-on-business

RealTime Machine Learning Challenges and Solutions. Chip Huyen January 2nd, 2022 https://huyenchip.com/2022/01/02/real-time-machine-learning-challenges-and-solutions.html

Appendix

Table 1. Loxz Digital MLR sub-scores

• Data Preparedness (DP): Quantifies an organization’s ability to efficiently and effectively locate, integrate, and leverage business resources to achieve its machine learning objectives.
• Model Development (MD): Measures the frequency and strategy behind how an organization leverages its resources to construct machine learning models to be as accurate as possible.
• Deploying Models (DM): Assesses the infrastructure, scalability, and methodology an organization uses to integrate machine learning models into systems that are in development or already part of their existing technical infrastructure.
• Model Monitoring (MM): Provides a basis for understanding the approach an organization takes in leveraging technical resources to maintain, monitor, and retrain machine learning models that are in production.
• Business Value (BV): Represents the alignment of strategic initiatives and use of machine learning models to enhance one's business.
• RealTimeML (RT) : RealTimeML provides insight on a company's ability to service models in RealTime, such as online predictions. We have 5 questions correlated to RealTimeML as part of the ML Lifecycle. RT will be introduced in the first quarter of 2022.

Table 2. Loxz Digital MLR score and subscores dashboard

Contributors

Chen Song

Data Scientist, Lead Author

Yumi Koyanagi

Report Design, Graphics