Welcome to Nextcloud News, our contributor blog roll. Nextcloud contributors should ask to get added!
Welcome to Nextcloud News, our contributor blog roll. Nextcloud contributors should ask to get added!
The architecture of Nextcloud is a classic Web Application architecture. I picked this architecture 7.5 years ago because it is very well known and is proven to be scaled relatively easily. This usually works with off the shelf technologies like http load balancers, clusters of Linux webservers and clustered databases.
But for many years users and customers asked for ways to distribute a single instance over several datacenters. A lot of users and customers run organizations that are not in one office or sometimes not even in one country or on one continent. So how can the service run distributed over different hosting centers on different continents?
Until now there was no good answer for this requirement.
Over the years I talked with users who experimented with different approaches. Unfortunately, they all didn’t work.
If you talk to storage people how to solve this challenge they say: No problem. Just use a distributed storage system like Gluster, Ceph, Hadoop or other.
If you talk to database people and how to do this they say: No problem! Just use one of the cluster and replication systems for Oracle, MySQL, MariaDB and others.
The challenge is how this all works together. If a user changes a file in Nextcloud then it is necessary that the file is changed, potential encryption keys are updated, log files are written, database tables are changed, email notification are sent, external workflow scripts are executed and a lot of other things happen. All these operations that are triggered by a single file change have to happen in an atomic way. It happens completely or not at all. Using database replication and storage replication independently will lead to a broken state and data loss. Additional problems are that you need the full bandwidth and storage in all data centers. So there is a lot of overhead.
The Global Scale architecture, that we designed, is currently the only solution, as far as I know, which solves this challenge.
Additional benefits are that the storage and database and overall operational costs decreases because simpler, commodity and more standard components can be used.
Another benefit is that the locality of data can be controlled. So if it is a legal requirement that certain files never leave a certain jurisdiction then this can be guaranteed with the right GS Balancer and File Access Control settings.
So far I only talk about the benefit that GS breaks a service down into several data centers. This is only half of the truth. Nextcloud Global Scale can be used in an even more radical way. You could stop using clustered Nextcloud instances in general. Killing all central storage boxes and databases and move completely to commodity hardware. Using only local storage and local databases and local caching. This changes the world completely and makes big storages, SAN, NFS and object store boxes completely obsolete.
The Global Scale architecture idea and implementation were developed over the last year with feedback and ideas from many different people. Storage experts, Database vendors, Linux distribution people, container and orchestration experts for easy automatic deployment and several big customers and users of Nextcloud.
The main inspiration came out of long discussions with the people at DeiC are running the National Research and Education Network of Denmark. DeiC was doing successful experiments with a similar architecture for a while already.
At Nextcloud we are committed to develop everything we do as open source. This includes this feature. Also, if you want to contribute to this architecture then no Contributer License Agreement is needed and you don’t need to transfer any rights to the Nextcloud company.
More information about Nextcloud Global Scale can be found here: https://nextcloud.com/globalscale
Together with the availability of Nextcloud 12, we’re proud to announce a new architecture for scaling Nextcloud several orders of magnitude beyond its current limits, lowering costs for medium to large deployments and increasing the flexibility to spread data out between data centers. Named Global Scale, this new architecture has its first pieces included with Nextcloud 12 and further implementation will become available over the coming months.
Nextcloud is used by home users and small businesses as well as large organizations like universities, big companies or government agencies. Nextcloud 11 introduced significant performance improvements and combined with further work in 12 enables customers to deploy installations up to tens of thousands of users. Above that scale, however, federation has to be employed, requiring separate instances where users have to log in.
Global Scale was designed to overcome the limitations at large scale, but also to achieve other benefits. The main goals are these:
Achieve several orders of magnitude greater scaling. It is hard to scale the standard architecture to instances over hundred thousand users. The shared components load balancers, hosting center uplink, database, storage and cache will sooner or later become bottlenecks. Especially the database is hard to scale beyond a 4 node Galera Cluster which limits the number of users and files. Nextcloud Global Scale aims to support up to hundreds of millions of users.
Several big Nextcloud users raised the issue that scaling the storage becomes exponentially more expensive when dealing with a high number of petabytes. Software-defined storage and object stores are unfortunately not a solution in reality. One of our Technical University customers estimated that 60% to 80% of the cost of running a file sync and share service is caused by the storage subsystem alone. It would significantly lower the total cost of ownership if the storage could be distributed over several smaller affordable storage systems. Similar cost issues exist with other pieces of the infrastructure like load balancers and databases where free or cheap solutions are not sufficient at large scale. Nextcloud Global Scale will enable the deployment on commodity hardware and software, dramatically decreasing costs for large systems.
A frequent need for large Nextcloud users is distributing data over multiple hosting centers in different countries or even continents. This can be due to legal requirements on where data is stored, to increase performance by bringing data closer to users or for cost, security or auditing reasons. With the current architecture, the choice is between a data replication solution for storage which hardly deals with most of these goals nor is fast and cost effective, or running multiple separate instances requiring users to log in in different portals. Nextcloud Global Scale will hide the existence of multiple data centers, making its architecture entirely transparent to users.
Nextcloud Global Scale works by effectively removing the need for shared components in the existing architecture like the load balancers, hosting center uplink, database, storage, and cache. It uses multiple independent application servers, called nodes, each running on standard, inexpensive commodity hardware. Storage, database, and cache are running local on the application servers and no longer have to be kept in sync.
Nodes can be located in different data centers and be as small or large as any current Nextcloud instance. A sensible scale would be at least 2 machines, providing redundancy in case of hardware failure. The machines run a web server with TLS, Nextcloud, local storage, local database and local cache. The nodes use central remote logging and central authentication like, for example, LDAP. The nodes could be managed using a standard technology like Docker containers to ease deployment and maintenance.
The Global Site Selector (GSS) acts as a central instance that is accessed by the user during the first login, accessing it via the Web, WebDAV or REST. The GSS authenticates the user via the central user management like for example LDAP. It then looks up the node where the user is located in the lookup server and redirects the user to the right hostname. The following calls during the same session are done directly from client to the node.
The lookup server stores the physical location of a user. It can be queried using a valid user id to fetch the federated sharing id of a user. In some situations, it is important to limit queries to a certain IP space to avoid data leaks. It also keeps track of old federated sharing IDs. The lookup server stores additional data of the users like for example the required QoS metrics like storage/quota settings, speed class, reliability class and so on.
The Balancer runs on a dedicated machine, monitoring the various nodes and their storage, CPU, RAM and network utilization. It can mark nodes as online or offline and initiate the migration of user accounts to different nodes based on data in the Lookup Server like business or legal requirements, QoS settings or user location. If for example, a user would move from the US to Europe, the Balancer would initiate a migration from their data to an EU data center to improve the quality of service.
You can learn more details on our webpage about Global Scale.
Today, Nextcloud 12 is released and with it come the first components of Global Scale. It introduces federated activities so users can know what happened to their data, even if shared to another node; and we have implementations of the (stand alone) Lookup Server and a Global Site Selector app will be in the app store in a few days. Work on user migration has started and we look to federate comments and release the Balancer over the coming months.
There is much more to Global Scale which our experienced engineering team will continue to develop and implement, working closely with customers, partners and of course our community.
If you are interested in the scaling and cost benefits or would like to explore the legal consequences of being able to decide where data resides, you can get in contact with our sales team.
Nextcloud continues to deliver impressive innovations. With GS, Nextcloud drives efficiency for cost-effective private cloud deployments to service internal and external customers which can be utilized while extending the service on a large scale.
— Florian Hausleitner, Senior IT System Engineer Datacenter Services at Raiffeisen Informatik Center Steiermark.
Nextcloud offers an industry-leading fully open source solution for on-premise data handling and communication with an uncompromising focus on security and privacy and unprecedented scalability. Nextcloud brings together universal access to data with next-generation secure communication and collaboration capabilities under direct control of IT and integrated with existing compliant infrastructure. Nextcloud’s open, modular architecture, emphasis on security and advanced federation capabilities enable modern enterprises to leverage their existing assets within and across the borders of their organization. For more information, visit
Welcome to Nextcloud 12! After five months of development and three more weeks of intensive testing, we are proud to release the final version of Nextcloud 12 with all its features, fixes and security improvements. With it comes our announcement of Global Scale, which you can read about in our blog here.
First of all, a big thank you to our awesome community, who helped develop, test and deliver this release! It is nearly impossible to test extensively on all the different platforms and architectures Nextcloud is supported on, and your participation is invaluable.
Nextcloud 12 is taking on the next big challenge for file sync and share technology by integrating collaboration and communication capabilities. File sync and share were a big step forward for productivity, and the time has come to go further by providing users with real-time collaboration tools. Frank blogged on opensource.com about his vision on the next challenge and in this post we go into the details of new features!
File sync and share technology have been tackling a major challenge: make sure that you would have the latest version of your important files across all your devices, and that you could share those documents with your colleagues or customers. But these data exchanges lack one thing: context. Who edited this file? How do I share my feedback? This Nextcloud release delivers much improved real-time collaboration and communication and expands on the sharing and notification features of Nextcloud.
One major way in which many users work with Nextcloud is through the web interface. We made two major improvements:
You will find that anywhere you see a user avatar, be it in the sharing or comments section or even notification, you can click and contact that person directly. There were many other refinements in the Nextcloud user interface as part of our ongoing usability, performance and stability work.
The Files app, probably the most used part of Nextcloud, has seen improvements in various areas.
Nextcloud 12 brings many improvements to the integrated Video calls app. Here are the three main features:
Push Notifications. Nextcloud 12 supports instant notifications to users in the web interface as well as the mobile applications (coming soon!). Users are also immediately notified of new shares, comments and more. Administrators can send push notifications to users from the command line, enabling integration in other applications and tools.
With this release, individual file shares can be made writable, enabling you to share a DOCX file through a public link and immediately collaborate in a shared editing session with others with Collabora Online. If a recipient (re)moves one of the files, you can also find it in your trash bin and restore it if you need.
Last but not least, working with others was also improved across servers, federating activities so you can see changes on remote shares in your activity feed. We have done this by implementing the ActivityPub API developed by the W3C, as part of our work on Global Scale.
The Circles app brings basic private-social-network capabilities to Nextcloud. With this app users can create their own groups and share files inside those groups. Sharing on social networks outside of Nextcloud is also possible via the Social Sharing apps which let you directly share public links via Twitter, Facebook, Google+, Diaspora, and others. Moreover, the Chat app, delivering XMPP based chat, was improved in various areas including a new feature, making it capable of retrieving the history of earlier conversations from a XMPP server.
Administrators will enjoy more control over collaboration with a series of new apps, including:
Security is a key strength of Nextcloud, aligning us with the concerns our users have about privacy and control over their data.
We protect your data, first of all, by having a strict security process, reviewing design, code and running security tests. You can read about our security efforts on our website. Furthermore, we develop active and passive security measures in Nextcloud. Nextcloud 11 introduced better password handling, CSP 3.0 and Same-site Cookies support improvements and expanded brute force protection. In Nextcloud 12, we hardened security further and implemented the following features:
Last but not least, we run a successful Security Bug Bounty program, where we reward responsible disclosure with up to $5,000 for qualifying vulnerabilities. We’re very proud of our 8 hour response time and quick bug fix turn-around!
We introduced what’s new in Nextcloud 12 when we published the beta and you can read more in those blogs and watch the videos:
You can find Nextcloud 12 on our download page. You can follow the manual upgrade procedure or use our built in updater, either from the graphical console or the command line. Note that we do a staged roll-out: only 15% of the users on Nextcloud 11.0.3 on the Stable release channel currently receive an update notification for Nextcloud 12. If you wish to update with the updater even though there is no notification, you can set the channel to beta, reload the page and proceed to upgrade. After the upgrade set the channel back to stable!
You can use the updater from the command line as follows:
sudo -u wwwrun php updater.phar
www-dataon Ubuntu/Debian systems.
Guest post by Swapnil Bhartiya
As everything is moving to the cloud, so is storage. According to a survey by Markets and Markets:
The cloud storage market size is estimated to grow from USD 23.76 Billion in 2016 to USD 74.94 Billion by 2021, at a CAGR of 25.8% from 2016 to 2021.
However, there is one interesting pattern that we are seeing. Despite this growth, the market share of Dropbox is declining whereas both Google and Microsoft are growing. There are no clear indicators why Dropbox is losing to these two companies. But there is one pointer: both Google and Microsoft offer much more than simple remote storage and file sync. Google Drive and OneDrive are more than cloud storage solutions, they are collaborative platforms.
Last week I was covering DockerCon and OpenStack Summit and I met many executives and developers and asked some questions around remote file sync and storage service. One thing was abundantly clear. In a personal capacity they all use Dropbox like solutions to keep a backup of their files on the cloud or to be able to access across multiple devices. These files are meant for single consumption. But things change in the company set-up, they use cloud based storage solutions for one and only one reason – sharing and collaboration.
I talked to one of my friends who works at Dropbox and while I can’t disclose her name however she admitted that there is a growing demand for collaborative features from enterprise customers. If you look at the commercial Dropbox offerings, including Dropbox for Business or Dropbox for Team they both have one common trait and emphasis: sharing and collaboration. In fact collaboration is the first highlight of Dropbox for business as you can see in the image from their website below.
Companies don’t need files to sit on remote servers, aka cloud. Companies need these files to be shared within teams where they can collaborate on projects to eliminate the unnecessary to and fro of email and messages.
Collaboration goes beyond simply being able to work on the same files, it extends to additional features like online chat, calendar, notes and much more. Unfortunately, Dropbox lacks almost all of these features and that’s where OneDrive with its massive suite of Microsoft Office products and Google Drive with G Suites beat Dropbox.
Could that be a reason why more and more companies are picking feature rich solutions, at the same cost, instead of just remote file sync and storage services? Could be. Unfortunately, all three products that I mentioned above are non-free. Enterprise users have started to lean towards products that they can influence, products that they know are secure and do what they claim to be doing.
That puts Nextcloud in a very unique position. It started off as an open source alternative to Dropbox where users had complete influence over the future of the product. It evolved into a platform and has become something similar to Google Drive or OneDrive where you can add Office 365, Outlook, chat or Calendar like capabilities through third party projects.
But that’s not what makes it unique, what makes it unique is the collaborative capabilities that come with total regional control where you can actually restrict access to files based on the geographical location of team members. That’s something enterprise customers are going to value more as the political landscape around the world is changing.
Does your company care about collaborative features while needing complete control of regional control over collaboration? What unique needs do you have? Share in the comments below.
Nearly one year ago we introduced the Nextcloud bug bounty program offering a significant monetary reward for reports of security vulnerabilities within Nextcloud. Security Bug Bounties are a ‘security best practice’ followed by large organizations like Microsoft, Uber, Github, Twitter and Slack which we’ve embraced. We’re proud to offer some of the highest bounties in the open source software industry, rewarding responsible disclosure with up to $5,000 for qualifying vulnerabilities. We’re also proud of our 8 hour response time and quick bug fix turn-around! We thought that it is time to do a short recap on the progress of our program and invite more people to participate.
security is hard, and mistakes are just unavoidable
Despite our good security track record and many innovative security hardenings added to Nextcloud over the years the reality is: security is hard, and mistakes are just unavoidable. The largest IT companies with big, well paid and experienced security teams run bug bounty programs for this very reason!
However, we can make it as hard as possible for an attacker. We do that first by having a strong process aimed at writing secure code, training our developers to take security in account and reviewing designs in advance and the code itself after it has been written. Second, we secure Nextcloud pro-actively by introducing security hardenings which decrease the likelihood of a successful exploitation. By performing internal testing, we get the confidence required for shipping. And last but not least external testing such as via our bug bounty program on HackerOne gives us another set of hundreds of eyes looking over our code and potentially discovering issues within our software.
Something that especially sparks our interests are reports involving a bypass of security hardenings. After a report of a security issue, we perform a root-cause analysis and try to aim to mitigate problems of this category completely in the future. A recent example was, for example, us hardening our shipped jQuery library in addition to fixing the reported vulnerability.
As you see, running a bug bounty program is something you should take seriously to get the most out of it. It does not replace internal security expertise but rather augments it, providing opportunities to fix whole classes of potential issues at once.
In the last year, we have had reports by 358 different white hat hackers reporting 676 issues to us, averaging around 1.8 reports per reporter. As you can see, most of these reports have been done right after we announced our bug bounty program which took some more internal coordination to handle. Nowadays, we get a steady stream of around 5-10 reports a week.
Of those 676 reports, we acted on 77 unique issues which have been reported by 83 different reporters. The other 599 issues were not considered a security risk or either duplicate of existing issues:
From these 77 reports, 18 qualified for monetary awards as they were within the Nextcloud software while the others targeted our infrastructure which we excluded from our bug bounty scope.
In total we spent $5,083 on bug bounties, resulting in an average bounty of $282.
We are quite proud of our performance, our all time response time is eight hours and our all time resolution time is about one month.
Those numbers mean that after an issue got reported to us the reporter receives a feedback usually within 8 hours. In average the issue has also been fixed, reviewed, regression tested and finally shipped to Nextcloud users in about one month.
The bug bounty program would not be so successful with the dozens of skillful hackers participating in it. We would like to give a special shout-out to those top 5 five reporters in our program:
Those five people are just a small sample of all the hackers that helped us until now. We would like to extend our sincere thanks to every single one! Thanks to all of you for making the internet a more secure place.
If you want to be featured in our next bug bounty program update head over to our bug bounty program on HackerOne and start submitting vulnerabilities. We look forward to your reports!