We were in a quarterly business review meeting with the Director of Finance of one of our major customers. During the meeting, the Director informed us that the usability and performance of the Google App Engine applications that we had delivered for them were quite fast and responsive, while the performance of the dashboards of the ERP (a world famous one at that) that they were using, was performing very slow and took way to much time to show the required results. He wanted to show us the problem that he was facing, so he fired up the ERP screen and put some filters and clicked ‘go’ to show the dashboard!! The spinner came and the screen was loading…loading…….loading! So we moved to other topics of discussion while the dashboard could get loaded! After 10 mins or so, we completed the meeting and we were about to leave…took a quick glance of the screen and it was still…..loading ! 🙂 Case, proven.
Our application did not have the same load as the ERP, but still had several dozen millions of rows and 100+ terabytes of data to process to show the various dashboards and analytics. It was quite a data intensive application. With several 100s of users analysing their data each day of this business crucial application, how could we have made the application be responsive while still scaling dynamically – it was due to the choice of the Google BigQuery database that we had used as the underlying data source for analytics.
Over the last 3-4 years, we have been in love with the choice of BigQuery in some of our heavy data processing applications. Being fully managed, highly scalable and always giving results in seconds (most often between 4 to 8 seconds) on massive data sets with complex queries, it has allowed us to focus more time on the application features, rather than all the technical aspects of fine-tuning the database and dealing with the infrastructure challenges. Traditional RDBMS databases would not have fit the need. It was a key for our customer satisfaction.
The recent announcement of the private alpha launch of Google Omni, is exciting since it solves some customer real life problems for us. For instance, data from a Warranty Management application would be moving to an Incident Management application and visa-versa with strong interconnections between the data in the 2 different systems. Similarly there are strong connections between data in Appraisal applications and data in the organization’s Competency Management application and Bonus calculation applications. Often, due to history, one of the applications would be an application that we would have built in Google App Engine with Big Query as the analytical data source, but the other applications might still be running on other technologies and other clouds. This always limited the analytical capabilities that we could offer the customers to only data coming from our own applications and often not combine it with the data from applications running on other clouds (though it’s possible to do a periodic database synchronization, that is quite expensive and not real time).
Now with the Google BigQuery Omni release, we can combine data from applications running in different clouds and provide real time data analytics to users – and this can immensely help users to understand their data much deeper, faster and in real time (and not to miss the point that the IS team will love the cost optimization of this approach). This helps our applications that use BigQuery to transform from being just a data source to a true and optimized data lake.
BigQuery Omni runs on Google Anthos. Anthos is a game-changer in the inevitable multi-cloud strategy adopted by many organizations. During the launch, Anthos seemingly being priced at a minimum starting cost of $10000 per month for 100 vCPUs, was quite expensive even for multi-billion revenue organizations, unless the organization has 1000s of CPUs to manage in a multi-cloud setting. For scenarios where data from a few applications spread over multiple clouds need to be accessed in a data lake setting, the Anthos pricing was seen as a blocking point for cost effectiveness.
With recent changes to Anthos pricing (effective September 1, 2020) and aligning it with the same very attractive pay-as-you-use policy for much of its products such as the Google App Engine, Firestore, BigQuery, Google has made Anthos usage very much compelling.
With BigQuery Omni using Anthos, it’s a great move from Google to have made changes to the Anthos pricing policy to be a lot more attractive along with its BigQuery Omni release. With these changes, the BigQuery beta release might be one of the most exciting launches to look forward to in the coming months.
