Dragonfly - An Open-source P2P-based Image and File Distribution System

Dragonfly is an open source intelligent P2P based image and file distribution system. Its goal is to tackle all distribution problems in cloud native scenarios. It is efficient with P2P and CDN technology which reduces image distribution time significantly and expedite delivery. it supports large-scale file distribution with improved downloading speed and success rate and lower consumption of bandwidth, especially cross-IDC bandwidth.

Its features include:

• Implement P2P files distribution with various storage types (HDFS, storage services from various cloud vendors, Maven, Yum, etc.) through a unified back-to-source adapter layer.
• Support more distribution modes: active pull, active push, real-time synchronization, remote replication, automatic warm-up, cross-cloud transmission, etc.
• Provide separation and decoupling between systems, scheduling and plug-in CDN. Support on-demand deployment with flexible types: light or heavy, inside or outside, to meet the actual needs of different scenarios.
• Newly designed P2P protocol framework based on GRPC with improved efficiency and stability.
• Perform encrypted transmission, account-based transmission authentication and rate limit, and multi-tenant isolation mechanism.
• Bear more efficient IO methods: multithreaded IO, memory mapping, DMA, etc.
• Advocate dynamic compression, in-memory file systems, and more efficient scheduling algorithms to improve distribution efficiency.
• Client allows third-party software to natively integrate Dragonfly’s P2P capabilities through C/S mode.
• Productivity: Support file uploading, task management of various distribution modes, data visualization, global control, etc.
• Consistent internal and external versions, shared core features, and individual extensions of non-generic features.
• Enhanced integration with ecology: Harbor, Nydus (on-demand image download), warehouse services for various cloud vendors, etc.