Nextcloud releases Assistant 2.0 and pushes AI-as-a-Service

Today, we bring you a major update to the Nextcloud AI Assistant with Context Chat, Context Write, GPU acceleration, the ability to split off the Nextcloud server & AI server, and MUCH more! And for those without $100K GPU’s (you know who you are), we have also incredible news: we are working with several prominent European hosting companies, including IONOS, OVHcloud and plusserver to deliver AI-as-a-Service solutions that respect your privacy and digital sovereignty! For our US users and customers, there are already several companies offering AI-as-a-Service and some we support (OpenAI in particular) while others are in the works.

Introducing Assistant 2.0

With a nearly complete overhaul of the user interface AND how our AI Assistant works on the back-end, plus a ton of new, big features, we decided that this is the moment to call this the Assistant 2.0. Let’s go over what’s new, but first a short summary of where we are today!

Nextcloud has a ton of AI features, some big and elaborate, others more basic. Some train a neural network on your data, on your own server to give you a smart inbox in Mail or warnings about unexpected logins from Suspicious Login Detection. Others rely on a pre-trained network that can recognize faces and objects in photos, generate a transcript of your video call or modify text. We can also generate images, translate text and chat messages, and much more. Many of the features are accessible through our easy to use Nextcloud Assistant interface!

Many of these features can help you, save you time, or improve your productivity. And best of all: as always we focus on making all these services available of running on your own server. Given even the US Congress doesn’t trust ChatGPT or Microsoft CoPilot, there are good reasons to want control over the data send to and from an AI service! Our Ethical AI Rating system provides you transparency about the AI used to generate your text, image or classify your photos:

• Is it open source? Can I study how it works and re-train or optimize the model for my needs?
• Is the model freely available? Can I run it on my own server?
• Is the training data open and available so I can inspect it for issues and re-train or optimize the model?

Learn more in our blog post about our approach to AI.

Context Chat and Context Write

And now, let’s explore what’s new: The two biggest new features are Context Write and Context Chat. These both share a lot in common, both providing AI-assisted outputs for you, but they’re also distinctly different.

Context Chat, in short, is the ability for the Assistant to answer questions about your data. You can ask the Assistant questions about a document you have: Say, you have a PDF with the manual of your new digital watch… you’ll be able to ask the Assistant how to set it up, or how to replace the battery. And if your company has a nicely documented reimbursement process in its documentation in Collectives, you can ask the Assistant questions about this process. This is an absolutely amazing ability!

It is not just limited to your files either: we developed an API so apps like Mail, Calendar, Talk and Deck can make their data available too. You’ll then be able to ask “When do I have my next meeting with my boss?” (from the Calendar) and “What tasks did she ask me to work on as high priority this week?” (from Mail). Or “Give me a summary of the status of project X.” (Deck + chat perhaps?)

Context Write, on the other hand, lets the Assistant rewrite something in a certain requested context, or style. Say, you have a nice poem and want to write another in that same particular style. By providing the original poem, then new text of what you want to say, the Assistant will then create a masterpiece. But you can also give it a form, give it your data, and ask it to provide the text you would need to fill in the form. Impressive, and useful!

Big user interface overhaul

Over the last year and a half, we have introduced a wide range of AI features in Nextcloud. Translation, transcription of audio, text and image generation, and more. Then the Nextcloud Assistant came – and we’ve now integrated most of our AI features into the Assistant itself, all available from a unified interface. It is no longer limited to text, but can also generate images and transcribe audio.

Another improvement many of you will appreciate is that you can now see a history, a list of earlier generated images, text or audio transcripts in the Assistant interface. You can even go back and generate more based on your earlier prompts. It is now also possible to cancel a running operation, if desired.

Lastly, as possible inputs to the Assistant you can paste or type text or record audio, and now… select one or more documents as input too!

Behind the scenes

We also did significant work behind the scenes. We made it possible for administrators to make use of AI models we’ve not yet explicitly integrated into Nextcloud, giving more flexibility and versatility to your choice of models to use.

Especially for larger deployments, we have made it possible to run the LLM and other AI operations like Transcription and Context Chat on a separate server, using our new external app ecosystem for one-click, Docker-based deployments. You can also now use a GPU for text generation, transcription and context chat. This means transcribing an hour-long audio file could, on a CPU, take many minutes while taking only seconds on a GPU!

More coming soon

And, of course, this is not all. We have developed an API which can be used by apps like Deck, Calendar or Mail to support Context Chat. This will allow the Assistant to answer questions about your mail, tasks or upcoming appointments! We look forward to these and other apps implementing support for this feature.

Nextcloud Hub 8 – our next version out very soon – will also feature two more app integrations of the Assistant: First, Nextcloud Talk will be able to show you summaries of conversation in Talk rooms, and Nextcloud Mail will introduce email ‘suggested answers’! If you haven’t yet registered for our Nextcloud Hub 8 Launch Event on April 24, this is the time to do so to find out more!

How to get it

So now, you ask: When can I get it? We have good news! Nearly all these features have been backported to Nextcloud Hub 7, meaning they are available in the Nextcloud app store right now. All you need to do is update. If you have not yet tried out the assistant, perhaps this is the time to do it!

To run the AI locally, it’s easiest to use LocalAI. A community container exists for our All-in-one, so users of the AIO can simply go to this page to learn more about adding extra containers and then enable the LocalAI container. Be sure to check out our documentation and ask questions in the forums – you could just start below this post!

IONOS, OVHcloud, plusserver and others bring you…

This brings us to the final piece of the puzzle: For these AI features to work smoothly, you will need some serious hardware. This unfortunately will not be a fun experience on your Raspberry Pi. But! Luckily, a number of businesses have started to provide AI-as-a-Service, running various open source models that Nextcloud supports in a way that you can connect to remotely. While this is great, unfortunately nearly all these companies are American-based, with a few in China. If you live in the USA, that is of course perfect – the data stays in your jurisdiction, and you have a choice of local providers. But for our users in Europe, this is less than ideal. The US jurisdiction unfortunately treats the data of European users with less than ideal respect for privacy, in no small part thanks to the CLOUD act signed into law by the former US president Donald Trump.

But today, we announce that we have been working closely with several major European hosting companies including IONOS, OVHcloud and plusserver to integrate their upcoming AI-as-a-Service solutions. This will make them available to Nextcloud customers and users who wish to take advantage of the advanced AI functionalities in Nextcloud Hub but don’t wish to host AI services themselves. Several Nextcloud customers have expressed interest in these features and are preparing to or in the process of testing them. Here’s what our partners and customers have to say:

OVHcloud is thrilled to collaborate with Nextcloud in delivering state-of-the-art Digital Sovereign AI solutions to our customers. We are aligned with Nextcloud’s commitment to offering secure, privacy-conscious AI functionalities that safeguard training data and adhere strictly to GDPR regulations. We are excited to announce that this integration will become available in Q2 2024, marking a significant milestone in our journey towards responsible AI innovation.

Germain Masse, AI Product Marketing Manager at OVHCloud

We are proud to offer Nextcloud customers a solution that complies with European data protection regulations. Reliable, trustworthy, and local AI services will be key to protecting their digital sovereignty for both the public and private sectors in Europe.

IONOS

plusKI brings cutting edge, open source AI technologies in a compliant and digitally sovereign service to the German market. We are excited to work with Nextcloud to make this available to their customers as back-end to the new generation of the Nextcloud Assistant.

Christian Schmitz, Director Open Source at plusserver

We at SUNET are happy that that Nextcloud Assistant is 100% Free and Open Source, runs on-premise and provides useful features that we hope will benefit our users going forward. We enjoy the very productive collaboration with Nextcloud.

Mike Nordin, Sunet, the Swedish University Computer Network

In the future, we will make AI services available to colleagues in the state administration directly at the workplace. We want to drive the development of digitally sovereign and open solutions in addition to the use of existing AI services such as ChatGPT. Because, the administration of the future will work in an automated, algorithmized, cloudified and data-based way. To make this vision of the future a reality, we must provide the appropriate tools. This also creates value and jobs in our domestic digital economy.

Minister of Digitisation and Head of the State Chancellery Dirk Schrödter, Schleswig-Holstein, Germany

US technology firms have a large head start on European companies offering large language models and other AI services to smaller firms and end customers. There is a significant number of offerings either using their own models like OpenAI, or providing one or a choice of open models like Llama, Gemma or Mistral as a service. In Europe, Mistral and Aleph Alpha are still working on offering their own models while only a few large hosting providers are looking to offer various open LLM’s in an AI-as-a-service model.

At the same time, there is a growing demand from our customers and users for European AI services. Running AI locally is expensive, and AI-as-a-Service from foreign tech firms is not a solution for most Nextcloud customers in Europe, given the privacy and sovereignty risks inherent to shipping large amounts of crucial data overseas. For this reason, we have been working with a handful of providers to offer their AI services. To speed up the process, we connected them with potential customers. As you can imagine, offering a service when it is unclear who its customers might be is not easy. But several universities are interested in the ability to outsource some heavy AI operations, so there is now movement towards AI services. This means not only our users, but the wider European ecosystem will be able to benefit from this!

Public availability of the European AI services is expected in the course of spring and summer, with some even aiming for late April/early May!

Thank you!

We are super excited about this release and we’d like to thank everybody who contributed, with testing, reviewing code but also the universities who are willing to test the AI services, and of course OVHcloud, IONOS, plusserver and our other partners who’d like to provide these services!