At the just concluded IGNITE 2019, Orlando, Florida, Microsoft CEO Satya Nadella, unveiled the Azure Synapse Analytics. The most advanced big data analytics service that aims to deliver a complete solution to data ingestion, preparation and management.
This could well be the platform enterprises are looking for to remove the primary impediment to wider adoption of AI. A thoroughly tested and proven solution to ensure the quality of the data amassed and analyzed to generate insights.
Designed to operate and process data at staggering speeds, AZURE Synapse Analytics will drive more and more businesses to reap the benefits of integrating AI & ML into their operations quickly and seamlessly.
We will be devoting an entire series of blogs to demystify Azure Synapse Analytics. The current blog however, is meant to be a primer to big data and AI. It will focus on making non-tech savvy executives understand the basics of Big data and AI. Hopefully, do this minus the hype.
So the first question, what is big data, really?
Here is a simple definition of big data. This one hits the nail right on its head. Big Data is a term used to describe data so huge and complex that traditional data tools are unable to store it or process it efficiently.
Why is Big data so important?
Big data can bring a whole new perspective to the way we handle data, take decisions and do business. But for that to happen however, a couple of things need to be done to data before we can use it.
To be called Big Data, data needs to be processed, and this step is critical because here is where things can go wrong. Data is like raw meat, when uncooked, it is inedible.
Depending on the data volume, the process of cleaning data is tedious and back breaking, even if you have automation to tide you over. But this is a very important step and you need to do it right the first time.
When you don’t clean, structure and integrate your data, you will have a surprise waiting for you every time you dive-in to retrieve your seemingly ‘authenticated’ data. You not only catch false data sets to work with, but also interpret them to derive misleading insights. Leading to mistakes that will cost you a lot, stumble into a deeper hole that will be hard to climb out of.
Let’s go back to Wikipedia and see what it has to say about the big “V’s” of big data:
Volume: Is the amount of data collected and stored. Its size usually defines the value and the potential for extracting useful insights from it. The quality of the data is what defines whether the data can be termed as big data or not.
Variety: The various types of data formats available, whether it is video, image, text, audio, from which big data draws upon to help analysts analyze and derive insights. Big data fills in the missing pieces through data synthesis.
Velocity: The pace at which the data is created and treated to fulfill the requirements to pave the way for business growth and development. Big data is often made available in real-time.
Veracity: The quality of the data defines the accuracy of the analysis. Enables enhanced agility in processing data and arrive at meaningful insights.
Big Data is primarily about helping you gain deep insights, and that becomes possible when you work only with accurate, authenticated and credible data. Having said that, let’s now dive deep to see how AI can make Big data work for us.
A fundamental problem with big data is figuring out how best to optimize it. Buzz words apart, the sheer volume of sifting data manually is painfully excruciating. Fortunately, this task is being taken over by AI.
AI is finding new and faster methods to make sense of data. Taking over the drudgery of query or SQL led insight discovery process. According to Steven Mih, CEO at Alluxio, “What used to be statistical models have now converged with computer science and has become AI and machine learning.”
While people will still play the critical role in managing and analyzing data, all that heavy lifting done in the past is being taken over by AI. What used to take for ever, can now happen in minutes, if not seconds.
Mathias Golombek, CTO at Exasol concludes, “For specific use cases, AI will revolutionize the way you get rules, decisions, and predictions done.” This may not apply to all questions but the essence of his insight is hard to dispute.
Moreover, CTOs are now seen applying similar principles – using AI to reduce manual, and labor-intensive tasks, which were earlier left to back-end boys and girls with their legacy systems to handle and to err massively.
An AI enabled machine assists you to analyze and interpret data sets and find the right insight to help solve the problem you are confronted with. AI can also do tasks previously handled by humans on its own, completing them faster and with reduced errors.
AI in analytics is all about understanding the process of decision-making and the ‘how’ of taking right decisions based on insights gained from big data. By addressing issues based on big data interpretations shared in real-time, AI can help resolve business issues on the go!
AI and ML are seen as powerful levers when it comes to big data. They give you a holistic view of all the millions of data points and provide the means to make connections between key data sets.
It is left to humans and enterprises to combine intuition with AI and enable machines learn from both data and humans to fulfill its role. Businesses that get it right will be able to share key insights with everyone who needs them to arrive at a right decision. Not like before, in the sense leaving it up to data scientists and business analysts to tell what you need to know and do.
This eliminates human bais, saves time and results in smoother business operations; faster insights garnered from big data will also improve business throughput.
Most discussions about machine learning converge and come back to data quality. The better your data quality, the higher is the value you can extract from it. Lower the data quality, and the value of it plummets to near zero. “If the data is dirty, any insights derived from it cannot be trusted,” says Moshe Kranc, CTO at Ness Digital Engineering.
Fortunately, Kranc doesn’t stop there in his assessment of this issue. “Fortunately, machine learning algorithms can detect outlier values and missing values, find duplicate records that describe the same entity with slightly different terminology…”. In other words, it is possible to turn everything old into something new and useful.
AI & ML have already opened up a new front. You can teach ML algorithms to make a decision or take an action based on forward-looking insights. “Today, AI is moving big data decisions to points further down the timeline, in more accurate ways, by using predictive analytics,” says Sean Werick, managing director of analytics at Sparkhound.
This is just like how we first learn to crawl before we start to walk. “The value to business increases with each progression through the analytics maturity model: beginning with process and data mapping, to descriptive analytics, to predictive analytics, and finally, to prescriptive analytics,” Werick further adds.
The combination of AI and Big Data is just beginning to reveal its true potential and what you can do by leveraging these emerging technologies. Future possibilities are predicted to be not just big but also have a game changing impact.
According to some pundits, most business applications are still showing their analog ancestry. People are seen pouring over endless reams of reports to discover information they can do something useful with.
To derive significant ROI from all the data enterprises are collecting and storing, CTOs have to connect individual AI/ML systems with all the other systems within their enterprises. Utilize intelligent software to make them communicate with and learn from each other. And then leverage the data they have to do work for their businesses, intelligently.
Even more impressive will be the ability to process large amounts of streamed data in near real-time and apply new techniques to analyze data for real-time insight sharing. This is the real starting point and not the end game of AI, Big Data & Big Insights.