Modernized Data Engineering

Modernized Data Engineering

post thumb

The Problem

The client is an enterprise with various number of data sources whose owners vary across different teams from network team to business application management teams. Adfolks put together a strategy and approach to centrally collate data from all the sources to a storage that is in the cloud and/or on-premise. This process of flow of data from origin to destination has subprocesses which leverage resources and tools with different skill sets and functionalities and management become very cumbersome. A business should ideally invest in talent and capital stages post data collection to create insights that will help advance the business.

The Solution

Because of the diverse platforms and platform changes that could happen in the future and the obvious associated management issues, it was always on the wish list of platform engineers to have a solution that could abstract the data flow process.[capture, ETL, Storage]. With kubernetes this is possible now to have a microservices and API architecture to do the desired job and scale out whenever required and scale back one’s the job is done. This can be packages as SaaS or PaaS or an on-premise license to best suit the need relieving the client on management headaches and enabling them to focus on the more fruitful analytics workloads. Also, there was significant cost benefit advantages as containers optimize CPU utilization.

The Benefit

  1. Easy on-boarding to data analytics & machine learning platforms which is tailored as per customer requirements. The same solution can be extended to business and marketing team to onboard and manage Wi-Fi data, PoS data etc.
  2. High automatic scalability as the data grows
  3. Customer does not need to have the technical expertise on running the platform
  4. Pay per usage