Our customer, a Californian company, working in the digital security domain, requested advancement of their data acquisition platform. While they already had a product collecting the same-type data from cloud storages, Secuvy AI wanted to advance their data-tracking abilities by extending the scope of the search. The new version was supposed to gather the data about its customers from various database platforms.
Creating a multi-database API connector able to dynamically collect, aggregate, cache, and store the data from MySQL, MongoDB, Postgres, Hive, and other databases to use it for the Data Engineering pipelines and processes within the product.
Accelerate the user’s data collection process, provided a corresponding request;
Help the business rapidly track user’s data in a system;
Help the business reassure its customers of GDPR-compliant handling of their data;
Automate the processes mentioned above.
When working on the project, numerous challenges stood in our way. Having applied the engineering-everywhere approach and with a bit of outside-the-box thinking, in less than seven months, we managed to:
Create adapters for different databases;
Create a server that can switch between databases on the fly;
Create AI-based graphs for database automatic query generation.
Given that customer's pivotal legal research - the platform's ultimate adherence to EU General Data Protection Regulation, the law's provisions underwent substantial research and analysis.
Enacted in 2018, GDPR supersedes the Data Protection Directive 95/46/EC, thus requesting in-depth research of the changes in the European law on personal data protection.
Following the legal research, the project's technical scope has been defined, allowing us to allocate a team and devise the stack for the project.
Drawing the product’s architecture was the next step meant to create a technical itinerary for the developers.
High-Level Architecture Structure
DB Connector Microservice
Query generator service
Pursuant to an extensive bulk of tech research, the project's development phase has been planned and implemented with precision.
Following a particular project-management strategy is crucial to the overall success of the product. At this stage, the development sprints have been established and the tasks have been distributed between the developers and the QA.
Our mission was to add additional features to a system that was not capable of dealing with a particular type of tasks.
Coding the Connectors
We have come up with connectors for MongoDB, PostgreSQL, and MySQL, and other databases.
Multiple Data Bases Selection Flow
DB Selection middleware
SSM credentials store
Request with DB in ctx
While dealing with the core of the project, we:
Used the architecture controller-service-repository pattern
Used ML on particular stages of development, resorting solely to API of ML algorithms
Managed to configure the data-type migration process, converting data from different databases, thus unifying it as a part of Data Engineering
Launched an on-going database query generation based on the graphs built on the data obtained from API machine-learning process generated on the samples from the very same databases
Let's See What We Can Do for You
Tell us about your project, and we will send you an estimate within 48 hours.