For the last couple of weeks I had been working on building an enhanced digital strategy for a leading Australian enterprise. We were looking at the aggregated enterprise-wide transaction data to identify opportunities to grow. I was unable to access the data directly and had to work with intermediary data engineers to get the details. Additionally there were times when the database were inaccessible or had resources dedicated for other activities. Overall we lost time everyday to access the data which impacted our quality of work.
Do Digital Leaders Feel the Same ?
I have had the good fortune of working with digital leaders for leading digital enterprises. They always looked beyond the dashboard to understand customer behavior and sending out questions on the insights from various data sources. Followed by a hustle across various teams to get data & the answers from various databases.
Last week I felt the same frustration as an exec or digital leader when you dont have the data for your strategy.
This weekend I solved it as a conceptual framework !!
What does ‘talking to Enterprise Data’ mean?
Talking to Data = Convert Natural Language to SQL Queries using AI & ML
In the last 10 years, a lot of enterprising minds have invested to solve this problem and building open source AI capabilities , creating opportunities for others. To keep this blogpost simple I want to quickly introduce our key friends in this space – Salesforce ‘s WikiSQL & Google’s TAPAS
Seq2SQL & WikiSQL by Salesforce
In 2017 Salesforce Research had submitted a paper titled Seq2SQL, a deep neural network to translate human language to SQL queries. Additionally they published WikiSQL, a data set hand-annotated examples of questions and SQL queries, distributed from Wikipedia. For any NLP based enhancements, WikiSQL is a must-have repo to train your models.
TAble PArSing (TAPAS) by Google
Google Research has leveled up the game with TAPAS, which predicts the answers based on the tabular data. The output of TAPAS is not SQL queries by analyze DB table as input and then select table cells based on pre-training over millions of tables and related text segments crawled from Wikipedia.
Why Snowflake ?
I see the following advantages of using Snowflake for this framework :
- Virtual Tables : From my experience, data-driven discussions are related to specific minor data points leading to SQL operations to create multiple tables. With Virtual Tables in Snowflake, you can create as many tables as you want as long as there are no edits/changes made
- Python Ready : Snowflake has built in connector for Python already available, so no need to build integrations from scratch for every new enterprise initiative
- Resources Ready to Scale : Snowflake is a Cloud Data Platform with the capability to add or remove resources based on our data needs. So no one has to wait for the data !!
How would User’s question be collected ?
Powered by a Universal Comms Framework users can connect with Snowflake using any audio interface and/or as a chatbot within Slack or MS Teams
The Universal Comms Framework would be a serverless solution that connects with the Snowflake DB when a user says ‘Hello Snowflake’. It would enable omnichannel capabilities by making its endpoint available to chatbots and audio interfaces to capture the users question or ask with the data.
How is User’s question transformed to SQL?
Before accepting user queries, I built a model using WikiSQL’s dataset and Coarse2Fine model with sample input questions. The end model would be able to take user’s question in human English language and then predict closest query possible in SQL format.
For more details you can view an estimate implementation in my GitHub here : https://github.com/sagarsea/text2SQL
How is the User’s question answered?
Using Snowflake’s Python Connector, the SQL query determined is executed to the correct Snowflake table.
At this time, I am proposing that the response from Snowflake be presented to the user as output of their query.
The Comms Framework does the job of connecting with the right Snowflake table and based on the response presents it back to the user. It is further changed into respective format based on the channel used such as audio
What are the known limitations ?
- Assumes that transactional data are relational and is not inline with global trends to move towards NoSQL
- Requires a cloud based real time database kike Snowflake and cannot integrate with other database engines or frameworks as available.
- Frequently asked questions are not stored which can have a big optimization impact on the comms framework