How Sentry helps Nextcloud build reliable and secure software

published first on sentry.io
Sentry is open-source error tracking that provides visibility across the entire stack, giving developers the details they need to fix their bugs. Even the most bumbling, typo-prone developer can use the Sentry service to resolve problems, well before their users encounter them. Sentry lets users track, prioritize, identify, reproduce, and fix production errors across every application in their stack.

Nextcloud Sentry integration

The Sentry app for Nextcloud keeps track of unhandled errors and exceptions from the Nextcloud core and any apps running on it. For that, both the PHP and JavaScript SDKs are used to hand over all crash information to Sentry.

With the integration, errors are collected and aggregated from Nextcloud in the Sentry web interface.

How it works

Nextcloud registers a global error handler in PHP that captures all unhandled errors. Errors are then passed to the central logging service. Apps can also acquire an instance of the logger to log any errors, warnings, and debug information.

In order to register crash reports, we needed to adapt the logger implementation. With this adaptation, when an error occurs, the logger adds the information to the system log file, and also invokes the crash reporter callbacks and passes some context information. In this case, Sentry is the crash reporter, and data is passed on to the Sentry SDK.

To enhance Issue reports on Sentry, the Nextcloud logger passes logging information (debug, info, warnings, etc) to the crash reporters. Sentry then uses this information to build Breadcrumbs.

On the client-side, the Sentry browser client does its magic and hooks into the error handlers of popular frameworks. In addition to the captured console logging statements that Sentry shows, the Breadcrumbs generated from the Nextcloud logger displays automatically.

How the integration helps developers at Nextcloud

Unlike other companies using Sentry for error tracking, Nextcloud is not directly operated by the team that develops the software. Instead, the software is hosted on premise by Nextcloud users.

At Nextcloud GmbH, engineers use Sentry for their personal Nextcloud instances as well as the company instance. In the past, this arrangement has helped find issues in betas, RCs, and stable releases that might have not been noticed otherwise.

Sentry treats app IDs automatically assigned by the system logger as tags on its user interface, making it possible to filter tickets for specific apps. We find these tags especially useful when maintainers of subsystems want to check if errors were triggered by any of their maintained apps.

Because Nextcloud passes Sentry release data, Sentry knows the Nextcloud version and can detect regressions. This means an issue will be automatically re-opened if it had been closed as resolved in a previous version. We often rely on this feature when testing pre-releases and going from Alphas to Betas to stable releases: should a known bug be triggered after it was marked as resolved in the previous release, Sentry will automagically re-open the ticket and warn about the regression.

Hunting bugs

When Sentry signals a new issue, one of the developers will triage the report and assign the developer who might know how to address the issue (most likely because the bug was triggered by an app they maintain). This is where the app ID, the stack trace, and Breadcrumbs help a lot.

The developer can then use the information from the report to try to reproduce the issue. Once it’s confirmed and fixed, the issue is marked as resolved. This will silence Sentry alerts until the next release. Should Sentry see the issue again, it will re-open the ticket and send another alert about the regression.

At Nextcloud, we saw most new/unknown issues from the front-end, as errors in the back-end of the application are usually caught and logged already. Errors on the front-end are often unnoticed, by both the admin and the users, and might cause issues for a long time without anybody ever noticing. Sentry has consistently revealed errors nobody had noticed before, ultimately contributing to Nextcloud’s increased stability.

Emphasizing security

Nextcloud puts a strong focus on protecting user data and security, hence it uses a strict content security policy (CSP). Unfortunately, as an admin, issues with CSP are usually unnoticed unless the users report them. Sentry, however, already has support for CSP violation reports, which will be integrated soon.

Sentry helps protecting user data with its automatic data scrubber. This tool automatically removes sensitive parameters like passwords and API keys from reports.