Distributed Bot Detection System

Creating sub 5ms Response Time analytical system which operates in multi-zone cloud environement with High Availability and Scalability for DDoS and Scraping Bot Mitigation

Industry: E-Commerce
What we did:
  • Research & Development
  • Architecture design
  • Backend development
  • Analytics
  • Cloudflare integtration
  • DevOps
  • Golang
  • Kubernetes
distributed bot detection system

Clutch.co, a leading marketplace for finding service providers, has been a key partner of ours since they first started. Known for connecting companies with the best solutions, they're always aiming to improve their services. Recently, they came to us with a specific challenge:

Design and development of distributed system which always respond in under 5ms, detects DDoS and content scraping Bots

Given our long-standing relationship, they trusted us to help them tackle this issue head-on.

In our rapidly evolving digital landscape, the need for robust defenses against threats like DDoS attacks and scraping bots has never been more crucial. These malicious entities can disrupt online services, compromise data, and harm user experiences.
Through extensive research and collaboration, we not only addressed this issue but also crafted an innovative architectural framework that has redefined our cybersecurity practices.

Meeting Stringent Requirements: Building a Cutting-Edge Distributed Bot Detection System

We have created software with the following characteristics after conducting a meticulous analysis of demanding prerequisites, which were thoughtfully outlined by our client:

Kubernetes as a go to operating environment

In the ever-evolving landscape of cyber threats, ensuring high availability is of paramount importance. Our system was designed to uphold unwavering resilience in all scenarios, effectively eliminating the possibility of downtime. Even in the face of unforeseen disruptions, our system remains a steadfast guardian, providing uninterrupted protection against malicious actors. To reinforce this robustness, we seamlessly integrated native Kubernetes features such as load balancing, vertical and horizontal autoscaling, and pod discoveries. These integrated capabilities empower our system to dynamically adapt to shifting conditions, further augmenting its capacity to deliver continuous service and safeguard against emerging threats, even amidst the dynamic challenges of today's network environments.

High Availability so we don't cause snowball effect

In the face of relentless cyber threats, achieving high availability was paramount. Our system was architected to maintain unwavering resilience under all circumstances, virtually eliminating downtime. Even in the event of unforeseen disruptions, the system stands as a steadfast guardian, ensuring uninterrupted protection against malicious actors.

Multi-Zone Resilience and Fault Tolerance

In the contemporary digital landscape spanning multiple geographic zones, the necessity for high availability and proximity to application components within this distributed environment led to the architecting of our system. This design incorporates an embedded distributed key/value store for efficient data and service management across boundaries. In case of zone failure, the system efficiently reroutes traffic, ensuring uninterrupted service and bolstering its resilience. Data about recent suspicious activity is always replicated across all k/v stores and available for thread detection algorithms.

Sub-5ms Response Time

In the realm of threat mitigation, every millisecond holds significance. We rigorously optimized our system's response time to adhere to the demanding sub-5ms threshold. This swift response ensures that all components of the application remain unaffected by the security layer, preserving network performance and user experiences without compromise.

5ms badbot

In our tests, we achieved response times that were in micro seconds so they were three fold faster than the 5ms threshold. When considering the interconnected nature of cloud environments, with systems operating within a single region achieving 2ms ping times, we can confidently state that our system surpasses the capabilities of current cloud networking infrastructure. Moreover, we have designed the system to withstand saturation, ensuring that its performance remains consistent even under the most demanding conditions.

Real-time Data Replication

To empower our system with realtime threat intelligence, we implemented robust data replication mechanisms elevating replication mechanism built into distributed key/value store. This capability allows our system to remain aware of events from other agents in the network, facilitating rapid threat detection and response.

Instant Scalability

DDOS and scraping bots traffic patterns can change in seconds. Leveraging native Kubernetes features, our system was engineered to adapt and scale rapidly in response to workload fluctuations. Whether facing sudden traffic spikes or expanding network demands, the system adjusts in realtime to maintain optimal performance.

Tech Stack

Meeting Stringent Requirements: Building a Cutting-Edge Distributed Bot Detection System


Kubernetes provides a framework for running distributed systems resiliently, allowing for containerized applications to run across a cluster of machines without tying them to individual hosts. This means it can handle scaling applications on the fly.


BigQuery is designed for scalability, allowing users to analyze terabytes to petabytes of data with ease. It supports a serverless architecture, meaning users do not need to manage any infrastructure or servers.


Go, is an programming language developed by Google designed for efficiency, readability, and simplicity. Go offers fast compilation and execution times, making it an attractive choice for developing a wide range of applications, from simple scripts to large-scale distributed systems.


Looker is a business intelligence (BI) and big data analytics platform that enables users to explore, analyze, and share real-time business insights. It operates on a cloud-based infrastructure, making it easily accessible from anywhere. Looker integrates with any SQL database or data warehouse, such as Google BigQuery.


Cloudflare is a global network designed to make everything you connect to the Internet secure. One notable functionality Cloudflare offers is its IP blacklisting capability which we use in this project.

Cloud Logging:

Google Cloud Logging, part of the Google Cloud Platform, is a fully managed service that allows you to store, search, analyze, monitor, and alert on logging data and events from Google Cloud and Amazon Web Services.

Cloud Build:

Cloud Build is a fully managed continuous integration and continuous delivery (CI/CD) platform that automates the process of building, testing, and deploying software across multiple environments

Beyond its security prowess, our system was thoughfully designed to provide live statistics, with data efficiently stored in BigQuery. These statistics offer valuable insights into ongoing threat landscapes and system performance, empowering proactive monitoring and data-driven decision-making. Network administrators gained realtime visibility into the system's operations through Looker dashboards and event presentations, ensuring robust cybersecurity management. Additionally, our system automatically blocks each detected event on the firewall, providing an immediate and proactive response to threats.

shape orange
8 weeks


shape blue

requests per/sec

shape purple


shape green

response time

This comprehensive system not only represents a formidable defense mechanism against DDoS attacks and scraping bots but also sets a new benchmark for distributed bot detection systems. Its adaptability, speed, and realtime insights make it well-equipped to address the evolving cybersecurity landscape while offering robust protection and performance.

Our commitment to excellence and innovation ensures that our system not only addresses current threats but also stands ready to adapt to the evolving cybersecurity landscape. The system's development, based on Golang with an embedded key/value store, and its integration with Cloudflare, Kubernetes API, and BigQuery exemplify our dedication to creating cuttin-edge solutions that push the boundaries of technology.

No project is too big or too small.

Schedule a call with us

arrow forward