I became the CTO of Zafin in January 2023 to focus on product development after many years at IBM in a consulting role. Zafin was extremely well positioned to address the banking core modernization challenge I was learning about so I was excited about Zafin and wanted to contribute to what it was creating.

In my 30+ years of building and scaling systems, one of the core challenges is choosing the streaming data architecture. For most organizations, it can be a nightmare. You want one that is not difficult to set up, complicated to manage, or costly for the team. Based on my work at IBM, I was already familiar with Redpanda as a streamlined Apache Kafka alternative.

The developer experience, Kafka compatibility, simplicity of day 2 operations, and a responsive team were all reasons I was looking at Redpanda.

In this post, I explore my thought process and the pain points that ultimately convinced us to use Redpanda at Zafin for one of our most strategic initiatives: the development of Zafin Integrate & Orchestrate (IO). But first, let me explain what Zafin does.

Zafin: a FinTech driving banking modernization and transformation

Zafin is a SaaS core modernization and transformation solution that empowers the world’s top banks to stay ahead of the competition in the digital era. With Zafin, banks accelerate time-to-market for new products and offers, personalize products and pricing, enhance the customer experience while lowering the cost of change, and achieve tangible business and risk outcomes.

Traditional banks often rely on legacy technology systems implemented decades ago. These systems are outdated and not designed to meet the demands of modern banking, particularly in the digital age. That’s where Zafin becomes a trusted partner to banks as they leverage our platform to increase business agility and accelerate their core modernization journey.

The Zafin platform moves products and pricing from the legacy banking system to a modern cross-enterprise layer. This makes it possible to add products beyond the one-size-fits-all approach of the old banking systems. Some examples of our solutions include:

• Core modernization separates products and pricing from the core banking systems, allowing banks to innovate without the risky introduction of a new core system.

• Dynamic relationship pricing defines rates and fees automatically based on numerous client attributes, extending customer lifetime value and driving preferred behaviors.

• Quote to cash adds clear visibility into customer pricing, eliminates manual data input, and holds customers accountable for their commitments.

• Enterprise product catalog centralizes product information, enabling business users to manage changes with minimal or no IT intervention.

Today, we partner with global financial institutions with sensitive compliance requirements and mission-critical applications, demonstrating Zafin’s value in automation, cost reduction, and product innovation. Our first significant milestone was introducing customized products and reducing processing times by more than 400% for Indian HDFC Bank. Zafin won a Retail Banking Project award for this project.

As we helped our clients with their core modernization initiatives, we realized we needed a real-time data integration and orchestration layer to easily connect the banks’ core systems with the Zafin platform. That led to Zafin building the Zafin Integrate & Orchestrate (Zafin IO) platform, which now has Redpanda at its core.

Enter Redpanda: building a reliable system of reference for real-time events

Back at IBM, I led the effort to build a streaming data platform for a large top-five North American bank. The streaming data platform would serve as a system of reference and be used for all read inquiries from channel applications.

We instrumented many systems of record at the bank to emit events in real time. Stream processing applications ingested these events into Redpanda and transformed them into a unified industry-aligned data model. This transformed data created materialized views for query-by-channel applications. We were also processing end-of-day batch files to enforce data reconciliation. At one point, we were processing 700-800 million events every day!

The benefits of this approach were significant:

• We broke down data silos. For the first time, we had all banks’ data in one place and had insights we could never have imagined.

• The channel applications all had a consistent pattern for getting data. They went to the streaming data platform and did not have to integrate with the many individual systems of record.

• The reads were offloaded from costly compute environments. This CQRS (Command and Query Responsibility Segregation) pattern requires low latency to avoid the read-what-you-write problem.

So, how did we get here?

Finding a Kafka alternative

Before Redpanda, we were running on Confluent and had 80 Kafka brokers in two regions, and it was never supposed to go down. Customer-facing applications depended on it. But it did go down — and more often than we expected.

As an engineer myself, I understand distributed applications. We had architected the solution to tolerate a certain number of brokers failing. We did not expect all brokers in both regions to fail. There was an outage where one broker failed and took down all 80 brokers for hours.

At that point, I began investigating why operating Kafka was so challenging and what other options were available. I read a piece about Kafka APIs without Kafka in one of the ThoughtWorks Technology Radars. Then I stumbled on a video of Redpanda’s founder — Alex Gallego — explaining the architecture of this streaming data platform built in C++. I was immediately hooked by the simplicity of its day 2 operations and the freedom from the JVM.

Testing Redpanda (and shocking our team)

All the claims sounded great, and in many cases, they are usually too good to be true. So I put a team together to put Redpanda to the test. We did a proof of technology focused on Kafka API compatibility and day 2 operations. The team basically took the bank’s environment, pulled Confluent out, and put Redpanda in — and they were shocked. It just worked.

Our Kafka streaming applications worked, our Kafka Connect connectors worked, etc. At one point, the team was sure that somehow they must be pointing to the Confluent Kafka and not Redpanda, so they removed Confluent entirely from the environment — only to find that everything still worked.

That was the game changer. We didn’t need to change our code, and suddenly, our day two operations were much simpler. We were used to having so many knobs to turn, but we hardly had to do anything with Redpanda. Redpanda also has a fantastic team that is exemplary of how to provide support, which gives us the confidence to embark on modernizing some of the biggest banks in the world.

“Redpanda just works. We didn’t need to change our code, and suddenly, our day two operations were dead simple.”

Redpanda in action: processing 300 million daily events from one of the world’s largest banks

Zafin is a SaaS product, and as such, it has standard and stable integration interfaces. The banks had to do the heavy lifting to integrate with Zafin using one or more of these interfaces and build the integration and orchestration capabilities. We wanted to help the banks streamline this integration to shorten the time to value.

That is what Zafin IO set out to do. Zafin IO is a technology platform that facilitates and accelerates integration between a banks’ core systems and Zafin’s platform. Essentially, it serves as a real-time data integration layer for core modernization. Redpanda is at the center of the Zafin IO platform. Other software products include Apache Nifi, Apache Flink, and Temporal.

Our first implementation of Zafin IO was at one of the largest banks in the world. With global operations and serving over 70 million customers worldwide, this institution produces more than a terabyte of data every day — translating to more than 300 million daily events.

To seamlessly integrate them with Zafin, we leveraged the data integration capability of Zafin IO and built data pipelines in three stages:

1. Pre-processing: Apache Nifi retrieves files, decrypts, converts from EBCDIC to ASCII, and streams to Redpanda topics.

2. Transformation: Apache Flink transforms the schema, filters the data, performs a delta against the previous day’s data, and enriches the data.

3. Post-processing: NiFi packages the data and sinks it to the Zafin SaaS platform database for processing.

We used Temporal to orchestrate all of the processing. Our setup is self-hosted in Azure Kubernetes Service (AKS) and only needs five brokers in the production environment, with another five brokers in our disaster recovery environment.

Zafin IO resonates with our clients, as they see the need for it in any banking core modernization effort and the significantly shorter time to value. Inspired by this smooth sailing experience, Zafin is now dedicated to going all-in with Redpanda. Before the end of 2024, we’ll replace five brokers and five ZooKeepers for every client with just three brokers of Redpanda.

“We’re excited to have a single platform that makes life easier for my operations team.”

What’s next: Redpanda Connect for even simpler streaming data pipelines

Replacing our Kafka implementation with Redpanda is only the beginning. The simplicity of Redpanda is important to Zafin. As we continue to drive core modernization at banks and increase everything from revenue to customer experiences, we depend on a streaming data platform that just works. Banks aren’t willing to modernize if they risk downtime, poor security, or compliance issues.

The recent acquisition of Benthos by Redpanda and the launch of Redpanda Connect will allow Zafin IO to ingest data effortlessly from our banking customers' many diverse core systems of record. With 220+ connectors, we will be able to cater to far more options than we can today.

The maniacal focus that both Benthos and Redpanda have on keeping the developer experience and Day 2 operations simple makes me very excited about Redpanda Connect and how much more it will help me simplify the data pipeline construct in Zafin IO.

To learn more about Zafin, visit our website and our Engineering Blog on Medium. To get started with Redpanda, grab a free trial to get a taste of the simpler side of streaming data, or contact our team for a demo.