Sometimes an engineering problem isn’t obvious until you need to iterate on it. That’s exactly what happened when we started working on the next version of Smart Photos at TinderⓇ.
How Smart Photos Works
At Tinder, Smart Photos is an opt-in feature that reorders a user’s profile pictures in order to optimize outcomes when shown to other users. This order is determined by a VLM (vision language model), built by Tinder engineers to rank photos that boost likes, encourage more matches, and start more conversations. You can read more about the behind-the-scenes in our tech blog here, but to keep it simple, below is a quick visualization of how it works at a high level.
As you can see in the diagram, the VLM assigns a probability to each photo, which is then used to ultimately select the pictures most likely to spur engagement. This comparison runs on a graphics processing unit (GPU) based architecture.
Once a user picks their photo set, the relative rankings of the photos won’t change. The ranking will only change if a user introduces a new photo(s), which needs to be ordered relative to the existing list. So, this process of determining the photo order can be computed asynchronously, and served to the user from storage.
To accomplish this, we use an ApacheⓇ Kafka-queue-based worker. It consumes photo update events to recompute the comparisons between the photos and store the final order for later use. This way when a user is served to others, we have the photo order ready to go with very low latency.
Reusing an Old Worker for a New Problem
When we got ready to update Smart Photos, we looked at previous iterations of Smart Photos since the requirements for this new GPU-based iteration were similar. Once we configured the worker to align with the new version of Smart Photos, we began to test the load on the new system.
Once we tried to scale, we immediately noticed that the load on the new GPUs was higher than expected. There were abrupt spikes of traffic, and we were passing all that load downstream to the most costly part of the system with no attenuation.
Two Problems, No Time to Waste
Suddenly, we were faced with two distinct but related engineering problems: 1) minimizing GPU costs by using as few of them as possible and 2) maintaining good availability during peak user traffic.
This project involved a lot of cross-functional teamwork, and everyone was invested in solving these issues as quickly as possible. We brainstormed heavily, attacking the problem from all sides. One idea was to rate-limit incoming requests, slowing traffic down and retrying later. This would definitely work, but it sounded like exactly what the Kafka queue was already supposed to be doing for us. Rather than implementing that bandage fix right away, I wanted to dig into the root cause first. The team was receptive to spending more time on this, even though time wasn’t on our side. That’s one of the things I love most about working at Tinder: engineers are given the agency and ownership to dive deep.
Going Back to the Basics
I started with a simple principle: the most obvious questions often lead to the most insightful answers, but if and only if, you take the time to validate the assumptions behind them.
Why were we seeing these spikes at all? Kafka was supposed to guarantee once-only processing, holding off on the next record until the current one was fully handled. So why did GPU load track the traffic spikes exactly, with no bottleneck from processing time?
The answer, it turned out, was buried in our configuration. Kafka can absolutely guarantee that behavior—but only if it's actually configured to. And when I dug into our legacy consumer, I found it wasn't: the system was acknowledging records immediately upon receipt, before processing even began.
For the new system, which ran a bit slower since we depended on external hardware, we were completely overloading the downstream during spikes. Some ranking jobs crashed due to the high GPU load. Moreover, this meant that the guaranteed-once-processing Kafka rule was violated. If a message had failed to be processed correctly, it had already been acknowledged, so it just wouldn’t get processed. This violated our guaranteed-once-processing assumption.
Once I was able to identify the root cause of the behavior we were seeing—normal spikes in user traffic directly overwhelming the downstream system because of the way our consumer was configured—I could start formulating a solution.
The Fix: Rebuilding on Spring Kafka
There were plenty of configuration options I could have tried, but one piece of technical debt stood out as the real target. Our Kafka consumer was built on an older, internal Kafka library, written years before Spring Kafka became the recommended way to integrate a Spring Boot application with Kafka. Spring Kafka is now widely used across the industry and integrates natively with our framework, and it gave us clean, out-of-the-box solutions to the two problems I'd identified.
1. Acknowledge messages after processing, not before
Kafka's default auto-commit fires inside poll(), committing the offsets returned by the previous poll, regardless of whether those records were actually processed successfully. A crash mid-batch replays the entire batch, and replaying up to max.poll.records means duplicate downstream calls for a single failure.
This was exactly the behavior baked into our legacy consumer's older internal library. Spring Kafka defaults to managing commits via AckMode rather than relying on Kafka's native auto-commit, but since we wanted wanted to minimize unnecessary retries and limit load on downstream services, I disabled auto-commit and committed per record instead:
With this in place, the listener signals outcome through control flow: returning normally will commit the record, while throwing an error hands it to the retry pipeline described below. AckMode.RECORD commits per record; AckMode.BATCH is cheaper but still replays the whole poll on a crash.
2. Configurable retry support
The second piece was retry behavior. Spring Kafka's RetryTopicConfiguration gives us non-blocking retries and a dead-letter topic with no custom code required. Instead of overwhelming the downstream during high load, a failing record gets republished to a retry topic to be tried again after a backoff period, as shown below:
Ordering isn't critical for our use case—we're processing photo upload events for different users—so this tradeoff was an easy one to accept in favor of deferring the retry load to a later time.
We also set initial back-off based on our downstream's actual recovery time. Retrying after a few seconds against a service that's still scaling up just produces more failures.
Records that exhaust every retry attempt land in the DLT, or Dead Letter Topic—a holding area for messages the system couldn't process even after retrying. A DLT handler catches them there.
This can be used to build whatever handling is needed, but it should be treated as a last line of defense: a record that lands in the DLT is permanent data loss unless something actively surfaces and addresses it.
A Calculated Bet
Exploring the above solutions took time digging into Kafka libraries and building a case for taking the time to rewrite the Kafka consumer with the above configurations in Spring Kafka. It was a timeline risk. I didn’t know for sure that it would solve the problem we were facing. But what I did know for sure was that it put us in a better position if the problem persisted. We would have less factors to debug, and a better baseline to build on.
The Payoff
This investment ended up paying off almost immediately. As soon as the worker with the new Kafka consumer was deployed, armed with its retry topics and new acknowledgment strategy, we saw the traffic stabilize and error rates sharply decline. For the first time, we saw the Kafka lag build up as the system waited for existing requests to process before working on new ones, which kept the downstream GPUs just the right amount of busy.
On top of saving us money in GPU costs, this stabilized the system so that we would no longer get paged if there was a sudden traffic spike and avoided manual increases to the GPU allocations. It also unlocked future iterations on a GPU-backed Smart Photo algorithm.
Why This Matters
The rate limiter solution that was first proposed would have made the spikes go away, but it wouldn't have told us why they were happening. That root cause would still be sitting there, ready to show up again somewhere else down the line—and it's the kind of shortcut that Tinder’s culture pushes back against.
We are always trying to move fast to keep up with our users. But we never lose sight of the value of getting things right. Engineers are empowered to challenge assumptions and given the trust to actually chase that answer down, instead of settling for a fix that just patches the symptom. The fixes may look the same from the outside, but being given the time and agency to address the root cause leads to lower maintainability risk and unlocks future iterations. It's the kind of culture that makes the product better, for the people building it and the people using it.
Trademarks & Endorsements Disclaimer: Apache Kafka and Kafka are trademarks of The Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied.
