As part of our push to make Nextcloud the best solution for securing enterprise data, we’ve worked on improving the performance of Nextcloud on large installations like the hundreds of thousands at DeiC. With database load the main factor limiting scalability for many of our customers, we’ve worked to decrease the number of database queries for Nextcloud 11 by up to 85%! This enables customers to have more users on a single server, reducing costs. Users will notice this decrease in server load in reduced load times and up twice as fast data transfers with especially small files.
Speed and scalability
From customers, we often hear that at large scale, the database is the most important bottleneck for scaling up Nextcloud servers. It is not easy to scale up databases as they eat lots of ram and CPU and while we’ve worked on using Redis for caching work results extensively it can’t take away all database work. Our focus has been on reducing the number of separate database calls, making sure we grab the data we need in one go rather than going back multiple times. For this release, the most significant change has been to reduce the number of calls related to getting information about shares to a user.
Shares and storage
Looking across a number of our customers we have concluded that the typical Nextcloud user has between 10 and 50 shares. As shares cause a significant number of database queries with every check for changes on the server, this was our focus. The consecutive changes resulted in a significant decrease of the number of queries and the time they take to execute.
Below some statistics of a ‘depth 0 Propfind’, one of the most common operations on a Nextcloud Server. This is the operation clients use to check if there have been any changes and they create up to 99% of the database load on a Nextcloud server!
Less database queries not only results in lower server load but also faster results. We have tested the effect on small (10 byte) file upload and download operations. This represents an optimistic scenario for these improvements as, with large files, Nextcloud simply saturates the bandwidth of a connection. But uploading small files is a significant bottleneck and this has become much faster.
Test setup and generalization
All tests were ran against a server with a local MariaDB database and using Redis as cache. As such a local database has a very low latency and very good performance, it is to be expected that on large installations the advantage of Nextcloud 11 over 10 will be even larger. Of course, as the graph shows, it depends heavily on the number of shares users typically have. If you would run this test on a huge instance where users have dozens to hundreds of shares accumulated over years of collaboration with others and provided the database is indeed the bottleneck in scaling up you can expect this system to easily handle perhaps an order of magnitude more users with Nextcloud 11.
The changes to database scalability are one among a number of enhancements meant to more efficiently use Nextcloud server hardware. Other enhancements include:
To make Nextcloud systems with object storage more scalable, this release introduces a multi bucket object store. Not all s3/swift implementations like having multiple million objects in a bucket, so Nextcloud 11 will create a bucket per user, helping with storage performance on large installations. When the storage system used doesn’t like to have too many buckets (Amazon S3), the number can be limited.
Thanks to improvements in the integration with Collabora Online, startup time has been improved in Nextcloud 11 and users will now be able to enjoy their online office suite faster than ever!
Large folders will load considerably faster due to optimizations in the Files app
Today, US-based file sync & share vendor Kiteworks announced their acquisition of ownCloud and Dracoon. Kiteworks points out that their customers now have access to their file-sharing application. It is to be expected they will not maintain 3 similar products, but customers will have to migrate to the US firms’ platform or look for another […]
As part of Schleswig-Holstein's state digitization strategy, the state chancellery has announced they will work with Nextcloud to develop AI for working with government documents. This comes just after we announced the first private AI assistant last weekend with Hub 6. The German state already uses Nextcloud and their AI strategy aligns with our work on ethical, local AI technologies.
Over the last year, AI has become a popular topic. Some is hype, some is substance. Some is good, some is bad. We want to give you the good, not the bad, and ignore the hype! AI has a ton of opportunity – but also risk. So we put you in control – off by […]
Our new Nextcloud Podcast host, Brent Gervais, had the opportunity to sit down with Max Schrems – privacy lawyer and pioneer in the European data privacy movement – to explore some thoughts and ideas that stemmed from Max’s keynote given at the Nextcloud Conference 2023. We explore the newly adopted Transatlantic Data Privacy Framework, it’s […]