Alibaba Cloud's Free Open-Source Database AI Algorithm PilotScope
-
Existing database systems are highly complex and demand extreme stability. Even integrating a single AI algorithm with a database for debugging requires engineers to work closely together for weeks or even months, resulting in inefficiency and poor outcomes. This has delayed the industry's ability to apply AI algorithms to databases.
To address this issue, Alibaba Cloud introduced PilotScope. By abstracting and generalizing modules and interfaces at the database and AI system levels, they developed a brand-new middleware system platform that enables "one-click deployment" of AI algorithms in databases. This technology comes pre-equipped with various AI algorithms, including mainstream database tasks such as parameter tuning, index recommendation, cardinality estimation, and query optimization. It has also been successfully adapted to two major open-source databases: PostgreSQL and Spark.
Experimental data shows that embedding AI algorithms into databases using PilotScope improves the speed of tasks like query optimization by 1 to 2 times compared to traditional "hard-coded" methods. Additionally, the overhead introduced by PilotScope itself is almost negligible.
Currently, PilotScope has been piloted within Alibaba Cloud, and the related technology has been open-sourced for free on GitHub and Modelscope community.
Project Address: https://github.com/alibaba/pilotscope