Shadow is probably best known for their remote gaming PC platform. Nextcloud-based Shadow Drive will allow users to store their data while keeping it accessible from anywhere.Read More
As part of our push to make Nextcloud the best solution for securing enterprise data, we’ve worked on improving the performance of Nextcloud on large installations like the hundreds of thousands at DeiC. With database load the main factor limiting scalability for many of our customers, we’ve worked to decrease the number of database queries for Nextcloud 11 by up to 85%! This enables customers to have more users on a single server, reducing costs. Users will notice this decrease in server load in reduced load times and up twice as fast data transfers with especially small files.
From customers, we often hear that at large scale, the database is the most important bottleneck for scaling up Nextcloud servers. It is not easy to scale up databases as they eat lots of ram and CPU and while we’ve worked on using Redis for caching work results extensively it can’t take away all database work. Our focus has been on reducing the number of separate database calls, making sure we grab the data we need in one go rather than going back multiple times. For this release, the most significant change has been to reduce the number of calls related to getting information about shares to a user.
Looking across a number of our customers we have concluded that the typical Nextcloud user has between 10 and 50 shares. As shares cause a significant number of database queries with every check for changes on the server, this was our focus. The consecutive changes resulted in a significant decrease of the number of queries and the time they take to execute.
Below some statistics of a ‘depth 0 Propfind’, one of the most common operations on a Nextcloud Server. This is the operation clients use to check if there have been any changes and they create up to 99% of the database load on a Nextcloud server!
Less database queries not only results in lower server load but also faster results. We have tested the effect on small (10 byte) file upload and download operations. This represents an optimistic scenario for these improvements as, with large files, Nextcloud simply saturates the bandwidth of a connection. But uploading small files is a significant bottleneck and this has become much faster.
All tests were ran against a server with a local MariaDB database and using Redis as cache. As such a local database has a very low latency and very good performance, it is to be expected that on large installations the advantage of Nextcloud 11 over 10 will be even larger. Of course, as the graph shows, it depends heavily on the number of shares users typically have. If you would run this test on a huge instance where users have dozens to hundreds of shares accumulated over years of collaboration with others and provided the database is indeed the bottleneck in scaling up you can expect this system to easily handle perhaps an order of magnitude more users with Nextcloud 11.
The changes to database scalability are one among a number of enhancements meant to more efficiently use Nextcloud server hardware. Other enhancements include: