D5 is a group of data scientists and engineers structured as a decentralized autonomous organization (DAO). They help startups, corporations, and organizations accelerate their data science & engineering initiatives. The project Blockchain-ETL completed by D5 facilitates data science on blockchain data. Available in Google BigQuery.
Building a high-scalable infrastructure for the Blockchain-ETL (available in Google BigQuery)
Business Task
The infrastructure of the blockchain-ETL platform was necessarily upgraded in such a way to maximize the automation of the deployment and maintenance processes, to make the infrastructure scalable and fault-tolerant, even in the most unpredictable situations and as quick as possible (within a few minutes).
The Dysnix team's task was:
- To optimize the current project infrastructure, create and implement a list of recommendations to achieve the highest level of fault-tolerance
- To describe the infrastructure as code
- TCO optimization (based on the work results, we managed to save 70% of costs)
- To achieve a high fault-tolerance of infrastructure and its security
- To ensure early warning of possible technical problems
The infrastructure was optimized following our recommendations. We described the infrastructure applying Terraform and Helmfile that allowed one to edit and add new components to it easily using the inheritance system, as well as quickly deploy new environments from scratch and effectively provide maintenance.
-
Solutions
- Building a high-available, secure, multizone, and scalable Kubernetes-based server Infrastructure in GCP with private topology
- Building a cost-effective infrastructure
- Full implementation of Infrastructure as code using Terraform
- Implementation of the monitoring system
- Building a logs aggregation system with a user interface
-
Value delivered by Dysnix
- Building a highly-available, secure, multizone and scalable server infrastructure for a blockchain-ETL platform
- All the development stages were shown as demos weekly so that customers could see progress in the work made
- Full integration of the Dysnix team into the project and the company not as an outsourcer, but a single in-house team, which is essential at the stage of implementation of the solutions
- Trained the in-house team on how to use the developed solutions
- Alongside them, the Dysnix team continues supporting the project