Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

The 2017 Apache Kafka Survey: Streaming Data on the Rise

In Q1, Confluent conducted a survey of the Apache Kafka® community and those using streaming platforms to learn about their application of streaming data. This is our second execution of this survey, so we do have some year over year information. With respondents from over 350 organizations across 47 countries, we’re feeling like this is a solid representation of where streaming data is today, and can be a valuable benchmark for understanding change in the data infrastructure market.

This blog introduces some key findings, or you can download the full report here.

86% of respondents are increasing their Kafka use, 20% by a lot

What will be no surprise to those who are active in the Kafka community: 86% of respondents are increasing their Kafka use, and 20% by a lot. At Confluent, we see this traction at conferences, meet-ups, in our Slack community, activity volume in online communities, and just the general excitement and stories when Kafka people get together. It’s hard to put your finger on that one thing that says a technology is creating a movement; it sure seems like that’s what’s happening here.kafka-growth

Over Half of Respondents are Using Kafka in the Public Cloud

It would be impossible for a trend with this growth to occur without making contact with the other huge trend in data infrastructure – the cloud! The survey indicates large organizations are moving to the public cloud with their Kafka clusters. In fact, over half of the survey respondents are using Kafka in the public cloud. Kafka can be used as a pipe to help companies move their data from on-premises systems to the cloud of their choice, eliminating vendor lock-in and assisting with the large amount of effort and coordination it takes to make that shift.

kafka-cloud

Apache Kafka: Not Just for Pub-Sub Anymore

We’re also seeing companies use Kafka for far more than publish-subscribe. The use cases for Kafka have grown to include microservices, IoT, and monitoring to name a few dramatic increases from the previous year. From those surveyed, two-thirds (66%) use Kafka for stream processing and three out of five (60%) use it for data integration. The most common use case for Kafka is data pipelines (81%), while half (50%) are already using it for microservices. As companies are adopting microservices for their distributed architectures, Kafka is here to help.

kafka-uses

Helpful tip: If you’re looking into microservices and want some general best practices, check out our resident expert Ben Stopford’s 3-part series here.

Microservices, Meet the Kafka Streams API

If you’re using microservices to build applications, it is made much easier with the Kafka Streams API as developers have the ability to work with data in-flight. Of the organizations that have microservices, already 28% use the Kafka Streams API to build them. The Kafka Streams API is most frequently utilized by developers (85%), but architects (48%) and application teams (43%) also use it. Introduced last year, the Kafka Streams API integrates into your streaming application to build and execute powerful stream processing functions, without needing to set up and run separate infrastructure. Despite its recency, most (89%) organizations are already familiar with the Kafka Streams API. The most common uses for the Kafka Streams API are ETL (40%), building core business applications (32%), and asynchronous applications (25%).

kafka-streams

Kafka Engineers are in Demand

With the growing adoption of Kafka, one conclusion worth noting is the lack of ability to hire Kafka talent. While we think Kafka engineers are generally pretty smart ( 🙂 ) turns out they’re also hard to hire. 75% of respondents said they are having difficulty finding Kafka trained engineers. We recently introduced a Kafka certification to help with that problem and also offer training to help developers get up to speed on all things Kafka.

kafka-skills

Join us at Kafka Summit NYC!

With these results and the growing use of Kafka and streaming platforms, we’re even more excited about Kafka Summit coming up next week. It’s the best place to learn from companies large and small doing some incredible work with Kafka. It’s not too late to register! We hope to see you there.

 

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

Real-Time Small Business Intelligence with ksqlDB

If you’re like me, you may be accustomed to reading articles about event streaming that are framed by large organizations and mountains of data. We’ve read about how the event […]

Sharpening your Stream Processing Skills with Kafka Tutorials

In the Apache Kafka® ecosystem, ksqlDB and Kafka Streams are two popular tools for building event streaming applications that are tightly integrated with Apache Kafka. While ksqlDB and Kafka Streams […]

Featuring Apache Kafka in the Netflix Studio and Finance World

Netflix spent an estimated $15 billion to produce world-class original content in 2019. When stakes are so high, it is paramount to enable our business with critical insights that help […]

Sign Up Now

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

Nouvelles inscriptions uniquement.

En cliquant sur le bouton « inscription » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Inscription » ci-dessus, vous acceptez les termes du/de la Conditions d'utilisation et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous comprenez également que nous traiterons vos informations personnelles conformément à notre Politique de confidentialité.

Gratuit à vie sur un seul broker Kafka
i

Le logiciel permettra une utilisation illimitée dans le temps de fonctionnalités commerciales sur un seul broker Kafka. Après l'ajout d'un second broker, un compteur de 30 jours démarrera automatiquement sur les fonctionnalités commerciales. Celui-ci ne pourra pas être réinitialisé en revenant à un seul broker.

Sélectionnez un type de déploiement
Déploiement manuel
  • tar
  • zip
  • deb
  • rpm
  • docker
ou
Déploiement automatique
  • kubernetes
  • ansible

En cliquant sur le bouton « télécharger gratuitement » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Téléchargement gratuit » ci-dessus, vous acceptez la Contrat de licence Confluent et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous acceptez également que vos renseignements personnels soient traitées conformément à notre Politique de confidentialité.

Ce site Web utilise des cookies afin d'améliorer l'expérience utilisateur et analyser les performances et le trafic sur notre site Web. Nous partageons également des informations concernant votre utilisation de notre site avec nos partenaires publicitaires, analytiques et de réseaux sociaux.