Search and AI-Driven Analytics, Architecting Automated Insights for the Masses

From driverless cars to virtual assistants, AI and machine learning have hit the mainstream. These smart technologies are also poised to make analytics easy and accessible for any business person.

SpotIQ AI-driven analytics automatically uncovers answers to questions business people may not have known to ask. Powered by an analytics platform with massive computing power, it learns what’s most important based on usage behaviour, spots hidden trends and patterns in the data, and delivers trusted and personalized insights to any business person.

Artificial Intelligence: Beyond the Hype

Artificial intelligence (AI) and machine learning have captured our imaginations for decades. As early as the 1950s, researchers created computer programs that mimicked basic intelligence as they interacted with humans, giving everyone wild optimism that machines would someday be able to perform tasks just like any human would. However, as expectations continued to rise, researchers were unable to achieve any meaningful results. Early computing lacked the processing power, speed, and scale required for machines to adequately mimic human behaviour. As a result, the initial euphoria around AI has met with scepticism that the promise was overhyped.

Fast forward to today and we are once again celebrating advances in AI, particularly in the field of machine learning. AI is no longer relegated to science fiction and Hollywood films. AI surrounds us in a variety of applications including computer games, autonomous vehicles, and virtual assistants. AI has become as ubiquitous as the enormous amounts of data we create to feed its underlying calculations and machine learning algorithms. With the support of seemingly boundless amounts of compute power, AI is applied in many consumer services and enterprise products today.

So what exactly are artificial intelligence and machine learning – and how are they different? In short, AI is a broad concept where machines behave and think more like humans. Machine learning is an application of AI that uses statistics and historical data to identify patterns and automatically improve at accomplishing a given set of tasks over time.

Machine learning is commonly classified into three different categories: supervised learning; unsupervised learning, and; reinforcement learning. With supervised learning, humans specify the desired outcome and manually classify a set of training data, and the machine learns how to classify new data to produce the desired outcome. Spam filtering is a good example of supervised learning. Spam filters learn from humans’ explicit classification of spam emails vs good emails. With unsupervised learning, the machine automatically determines how to classify data without human intervention, continuously adapting and improving its ability to accomplish a task all on its own. Playlist curation and content recommendations on sites like YouTube are examples of unsupervised learning as the system automatically learns each user’s preferences from his/her interaction with the content, without any explicit action from the user. And with reinforcement learning, the machine marches towards a particular goal and its behaviour as it navigates down its path is influenced by rewards or punishments such as a user’s explicit approval or disapproval via a thumbs up or thumbs down the action. All three categories can be interwoven.

Going one layer deeper, deep learning is a machine learning technique that processes data through multi-layered neural networks – processes inspired by the structure of the human brain. Deep learning algorithms are extremely powerful, producing desired outcomes by breaking down a problem into small chunks and crunching through large amounts of data at scale. As an example, in self-driving cars, one part of the algorithm recognizes cars in other lanes, another recognizes pedestrians, and another may even recognize street signs. All these pieces work together to help the car navigate safely to its destination.

Everyday Applications of AI
As consumers, AI is all around us. Consider this example. You roll out of bed in the morning and ask your favourite virtual assistant, “What is the weather today?” Popular devices such as the Amazon Echo and Google Home use AI techniques like far-field voice recognition to isolate your voice in a noisy room and then use natural language processing (NLP) to parse your question. Then, they recognize your location and use weather source APIs to respond with something like, “It is 77° Fahrenheit with clear skies in Memphis today.” This is an example of AI and machine learning built into virtual assistants that enable smart devices to act like an intelligent human.

When you open your laptop and fire up a web browser, your initial destination maybe your favourite search engine, such as Google. As you type your question in the search box, the search engine returns relevant suggestions in real-time based on every character you type. But not all search engines are created equal. Google’s search engine has become the gold standard because of the quality of the results it returns, driven by its sophisticated PageRank machine learning algorithm. PageRank uses the billions of documents on the web and data about the number and quality of links within a webpage to automatically determine how relevant content is based on your search terms.

Every time you upload a photo to Facebook, it uses facial recognition AI to pick out faces in the photo and automatically makes recommendations on who should be tagged in each photo based on patterns found in other photos of that person. In 2015, Facebook researchers rolled out the deep learning image recognition algorithm DeepFace that they claim to be 97% accurate. Similar technologies power Google’s image search and Apple’s facial recognition software that automatically recognizes your face to unlock your phone.

The list of AI-powered consumer experiences doesn’t end there, signalling how pervasive and mainstream AI has become. Naturally, there is a real excitement about AI and machine learning in the data and analytics community as this technology makes its way into the enterprise.

The AI Opportunity in Analytics

IDC forecasts the amount of data created annually at a staggering 175 Zettabytes (or 175 trillion gigabytes) in 20251. Compare that to 33 Zettabytes of data created in 2018. Connected people – 75% of the world’s population – will interact with data every 18 seconds on average in 2025, according to IDC.

Unfortunately, while data volume is rapidly growing, the volume of insights we’re able to extract from it has been fundamentally limited. That’s because in today’s analytics paradigm there’s a huge gap between data supply and data demand. On one end, there are many data consumers across every line of business who crave new insights. On the other end, there are a few data producers – the data experts – who are required to extract value from data. As more and more data is collected, this small group of trained experts is under more pressure.

The amount of data being created annually is forecasted to reach a staggering 180 Zettabytes in 2025

That’s why AI and machine learning present such a significant opportunity in the world of analytics. By infusing AI into analytics workflows, you can transform your organization and bridge the supplier-consumer divide by giving everyone access to the tools they need to make data-driven decisions. The good news is that AI has already arrived and is changing the way business people – such as marketers and salespeople – interact with the data they have at their disposal. Today, the uses of AI in analytics can be boiled down to three categories of technology: automated data discovery; search and text-based analytics, and; intelligent modelling and recommendations.

Automated Data Discovery
Automated data discovery encompasses a class of technologies that automates the process of data analysis and exploration in real-time. This includes everything from selecting data sets to explore, running queries automatically, combing through results for insights, and choosing a best-fit visualization paired with a natural language description of each insight.

The number of possible questions to ask of data is often too much for any human. With automated data discovery technologies, business people can rely on machine-driven smarts to explore complex datasets with a few clicks and get insights explained to them in natural language. They don’t need a trained analyst and hours it would otherwise take to explore the data manually and build a report. Instead, data experts can focus on data governance, building bulletproof data models, and preparing new datasets for analysis.

Search and Integrated Text-Based Analytics
AI-driven search is transforming the way business people interact with enterprise information assets. Unlike the search technologies of the past that allowed people to search for content within pre-built reports, AI-driven search makes it possible for business users to ask a question either by typing it into a search bar or via a voice-driven assistant to get answers to net new questions in seconds. Analytical search engines use AI techniques such as NLP to intelligently parse the question and transform it into a query designed for relational databases and machine learning to present personalized, relevant search suggestions in real-time.

Conversational chatbots for analytics expand on this question-and-answer paradigm by leveraging the virtual assistants and other devices to extend access to data outside of the analytics environment. Bots serve as a virtual liaison between humans and analytics engines and make it possible for business users to query and interact with their data on-the-go. These virtual assistants can be found within the modern instant messaging clients like Slack and native mobile apps, helping to provide critical insights and data context anywhere at any time.

Intelligent Modeling and Recommendations
Data experts spend a significant amount of time profiling data and modelling relationships between data sets to answer specific business questions. In fact, according to a 2017 survey conducted by FigureEight (formerly CrowdFlower), data experts spend 80% of their time just cleaning and organizing data for analysis. In the same survey, 60% of participants indicated that data preparation was the least enjoyable part of their work, making data preparation prime for disruption by AI.

AI-powered data modelling and recommendations can reduce time spent on this kind of work by automatically generating statistics about data sets, inferring data types, identifying hierarchies and relationships within data sets, and dynamically aggregating data at query-time. This enables the new class of citizen data scientists and frees up time for data experts to focus on the work they enjoy most such as data mining and more sophisticated analyses like predictive analytics to help the business stay ahead of the curve.

This covers just a few examples of how AI is fundamentally changing the world of analytics. Applied correctly, artificial intelligence has the potential to substantially impact or predict business outcomes, exponentially improve employee productivity and decision-making, and even create new jobs within the data team by increasing data literacy.

Machine-generated insights also help to minimize errors in analysis and eliminate human bias, bringing to your attention new metrics and business drivers that weren’t considered before.

Ultimately, the ease and speed with which new insights are detected to enable citizen data professionals to source, prepare, and analyze their data without the need for trained resources. But what will it take for your organization to adopt these kinds of AI-driven analytics technologies and change the way your business operates?

Trust-based AI
There’s a problem lurking at the core of AI in today’s world. Many understand what innovative AI technologies can accomplish but few know how they work, creating a general feeling of distrust. Business professionals count on analytics technologies to drive their most critical decisions, so they may find it difficult to adopt revolutionary technologies such as AI without understanding what’s happening under the hood.

And that’s why trust is the key to adoption of AI-infused analytics. When it comes to analytics, trust is created by delivering accurate, relevant, and transparent results. To do this, machines should not rely solely on their built-in learning algorithms but must work together with humans to ensure every result meets these standards of trust.

This philosophy underlies ThoughtSpot’s SpotIQ – an automated data insights engine that makes it easy for any business person to automatically generate trusted insights with a single click.

Source: ThoughtSpot