Nextcloud faster than ever! Introducing Files High Performance Back-end

At a virtual presentation streamed worldwide, the Nextcloud team introduced the availability of Nextcloud Hub 21.

Nextcloud Hub 21 brings a wide range of improvements in file handling performance with Files High Performance Back-end and important new collaboration features in Text, Talk, Groupware and Files. In this blog, we go into a bit more detail of the performance improvements.

For Nextcloud 21, we wanted to do something that benefits all users: make it faster.

To increase the capacity of especially large installations and keep Nextcloud snappy, our team profiled and optimized various core areas of Nextcloud. We made all kinds of improvements and you can watch our release video above to get some more details.

With the COVID crisis continuing to force employees to work from home, Nextcloud server administrators need to keep adding capacity. Nextcloud 21 will reduce the load on servers, enabling more employees to work without requiring additional hardware. At the same time, the new collaboration features improve the day to day productivity and support team members in getting more work done in less time.

— Frank Karlitschek, CEO and founder of Nextcloud GmbH

We did a number of improvements that reduce the amount of code loaded on each page, for example handling the translations and theming more efficiently. We also worked hard to optimize our use of the database, as this is an important limit for large deployments. Many common database queries were reduced or even outright eliminated. The move to PHP 8 will also give a nice performance boost for systems that have it installed!

Besides this generic performance work, we also looked at some specific scenarios where servers would run into performance limitations. Nextcloud Text, widely used in many organizations for note taking and basic writing, creates load on the server by checking in for changes constantly. We reduced this load by 25% and introduced a ‘idle mode’ where a document not used for 30 minutes stops polling the server. We also optimized object storage handling, made loading the trash view twice as fast, optimized LDAP, installation and upgrading, the app store and much more.

High Performance Back-end 🚀

The performance improvements above should impact nearly every server, giving a better user experience and enabling system administrators to grow the user base without having to add more hardware. But there is another, big change that we have been working on, and it requires a little background.

Nextcloud offers near-perfect scalability

By using the traditional LAMP stack with Linux, Apache, Mysql and PHP, Nextcloud builds on some of the most popular technologies on the web. This has many benefits. Our platform is easy to deploy and maintain and scales extremely well. The scaling, in particular, is a huge benefit of PHP over practically every other technology out there: to double the performance of a PHP server, you can simply have twice as many cores and things will work without any changes needed. Not only is the scaling you get pretty much 100%, unlike other languages, this scaling continues beyond a single server! You can put a second server next to it and let it handle requests. This is possible because PHP handles each request entirely independently from every other request. The result is that it is easy to create a cluster that offers Nextcloud to hundreds of thousands of users on a single instance.

The scalability of Nextcloud itself is thus near-infinite, but file systems and databases are not. The improvements we did in this release will help to scale further, but inevitably they will become the limiting factor. That is why we invented Global Scale! This innovation allows multi-continental deployments with hundreds of millions of users. We already deployed the first Global Scale setup in early 2018 and our largest customers use it with tens of millions of users today. Scalability is certainly not something Nextcloud lacks!

Taking advantage of new technologies

But every technical choice has two sides to the coin, and some emerging technologies on the internet provide other, unique benefits. To make sure our users always get the best of both worlds, Nextcloud 21 introduces a new, completely optional component: our High Performance Back-end (HPB) for Files.

The new HPB is a binary written in Rust and enables Nextcloud to keep a permanent connection open to our clients. So what impact does this have?

Faster notifications and syncing

First, you no longer have to wait! Currently, the client checks every 30 seconds to see if there have been changes on the server. With the open connection, the server notifies the client of changes and it can begin downloading new files right away. So when a colleague asks «did you get the file I just shared», you no longer have to say: «wait, it just started syncing!»

This does not only impact file syncing, however. It also means you get instantly notified of new chat messages in Talk or an incoming call. New files shared with you. A colleague commenting on a file. A Deck card running into its deadline. And much more! In many of these cases, the faster response time makes a real difference.

90% less load from polling

Server administrators will also be delighted. One check every 30 seconds equates to 2880 times per day. We reduced this number by 90%, only checking in once per five minutes.

To give you an idea of the impact of this change, we did an analysis of traffic at the German DFN Cloud, with more than 3 dozen educational and research institutes managed by the TU Berlin. In a 15 minute period, more than 250.000 requests are made by desktop clients just asking if there were any new files to download! This accounted for over 2/3rd of their network connections. You can imagine the impact of reducing that traffic by 90%. And that is just the desktop client – we also implemented this in the web interface, which also checks every 30 seconds! A user with 3 tabs open is polling the server 3 times every 30 seconds, 10 tabs means a ping every 3 seconds, all day long. You can imagine what impact it can have to reduce this by 90%.

At a scale of millions of users, milliseconds start to add up. After analyzing application and caching server, storage and database behavior, our team was able to significantly reduce the impact of common operations. The High Performance Back-end, on the other hand, provides a completely new way of reducing server load while bringing a new level of responsiveness to users.

— Roeland Douma, Server Engineering and Support Lead

The HPB for Files requires a reverse-proxy setup and Redis, among other pieces of infrastructure. You can find instructions here.

Read the full announcement about Nextcloud Hub 21 features and improvements!