Project Metamorphosis : dévoilement de la plateforme de streaming d'événements nouvelle générationEn savoir plus
July | Infinite

Infinite retention for Apache Kafka® with Confluent

  • Transform Kafka into a central nervous system for all events
  • Infinitely retain events without pre-provisioning storage
  • Reduce storage costs up to 70% in Confluent Platform

Infinite resources

July 1

Infinite Storage in Confluent Cloud for Apache Kafka

Learn what unlimited storage unlocks in Confluent Cloud

Aug 4

Infinite Retention in Confluent Cloud

See how to set up infinite storage and retention in Confluent Cloud

Why infinite matters

Transform Kafka into your central nervous system for all events

In traditional data architectures, storage systems focus on accumulating past records while messaging services are waiting to process future events. On top of this data layer, there may be hundreds of in-house systems, SaaS applications, or microservices both generating and consuming data. Data integrity and correctness must be maintained across all these systems, which is a huge operational burden and business risk as organizations and apps must make quick decisions on what they believe is the most accurate and up to date version of the data while retaining what is needed for regulatory compliance. Confluent's event streaming solutions allow you to:

  • With no limits on storage, retain all historical and current events in Kafka as long as you want
    • Set retention at the topic level to be as long as you want without scaling additional compute resources to handle the storage
    • Reduce tech debt and downtime risk with a central source of truth for all events across your entire ecosystem
  • Democratize access to your data with all events in one place and eliminate the need for complex data engineering
    • Maintain data compatibility with Schema Registry and enforce data integrity with Schema Validation to build an infinite and immutable central source of truth for all streaming events
    • Keep the flexibility to use your programming language of choice and transform infinite amounts of data in Kafka using clients such as C, .NET, Java, GO, Python and more
  • Reduce business risk by setting retention policies that help maintain regulatory compliance and providing a persistent log of all events for compliance audits
Infinitely retain events without ever worrying about storage

Provisioning massive Kafka clusters and retaining the right amount of data in the cloud with Apache Kafka is complex. Setting it up properly to handle GBps throughputs and petabytes of data can turn into months of work that push delivery timelines delaying business go-to-market. Similarly, businesses that don't need GBps scale or petabytes of storage at the beginning but face unexpected spikes or storage needs risk causing downtime or degrading cluster performance, which can lead to customer frustration. Confluent Cloud allows you to:

  • Never worry about Kafka storage again in Confluent Cloud with elastically scalable storage that automatically grows with your traffic
    • Avoid downtime and data loss by auto-scaling storage based on retention policies and traffic
  • Only pay for storage used with Confluent Cloud instead of paying for over provisioned storage volumes attached to Kafka brokers
    • Only pay for retained data, not pre-provisioned storage capacity
    • Decouple storage and compute to avoid unnecessary compute costs for high storage use cases
Reduce storage costs up to 70% with tiered storage

Storing historical data on self-managed Kafka clusters quickly increases your infrastructure costs because you have to over provision storage to prevent downtime and deploy larger clusters to avoid hitting storage limitations on brokers. It becomes cost-prohibitive to retain all the data you need in Kafka, causing organizations to glue together disjointed architectures that end up costing more money and siphoning off more engineering resources. Confluent Platform allows you to:

  • Send long term storage to cost-efficient object storage, such as S3 and GCS, using Tiered Storage in Confluent Platform
    • Expand cluster storage in Confluent Platform without any additional resource provisioning when you offload data to 3rd party object storage
  • Reduce partition rebalancing time for self-managed clusters from hours to seconds using Tiered Storage to minimize the size of your partitions on brokers

Benefits

Scalable Storage
Rétention infinie
Compatibilité des données
Confluent Cloud

Decouple compute and storage in Kafka to elastically scale storage

Infinitely retain events in Kafka only paying for storage used

Enforce consistent data formats for app development

Plateforme Confluent

Decouple compute and storage in Kafka to elastically scale storage

Infinitely retain events in Kafka by offloading data to object storage

Programmatically enforce data schemas at the broker-level

Fonctionnalités

  • July

    Infinite retention

    Store data in Basic and Standard clusters on AWS for as long as it is needed

  • Available

    Schema Registry

    Enforce consistent data formats to enable app development compatibility with Avro, Protobuf & JSON

  • July

    Schema validation

    Enable centralized policy enforcement and data governance within your event streaming platform

  • Available

    Multi-language development

    Enable all developers to leverage Kafka throughout the organization

  • Prévisualisation

    Tiered storage

    Retain infinite data in Kafka by offloading older topic data to cost-effective object storage

  • Available

    Schema Registry

    Enforce consistent data formats to enable app development compatibility with Avro, Protobuf & JSON

  • Available

    Schema validation for Schema Registry

    Programmatically validate and enforce schemas at the broker-level

  • Available

    Multi-language development

    Enable all developers to leverage Kafka throughout the organization

More Project Metamorphosis releases

Cost Effective

How do you keep the costs of running Kafka low and your best people focused on critical projects driving competitive advantage and revenue?

Lire la suite

Elastic Scaling

How do you quickly scale Kafka to keep mission-critical apps running with no lag or downtime - and without over-provisioning expensive resources?

Lire la suite

Everywhere

How can you ensure your Kafka infrastructure is flexible enough to adapt to your changing cloud requirements?

Lire la suite

Try it out

Cloud
Service entièrement géré

Déployez-le en quelques minutes. Payez à l'utilisation. Faites l'expérience de Kafka sans serveur.

Plateforme
Logiciel auto-géré

Découvrez la puissance de notre plateforme adaptée à l'entreprise grâce à notre téléchargement gratuit.

*Receive $200 off your bill each calendar month for the first three months