An internal tool when…

You need extremely fast Kafka message query, result returns around 200-500ms rather than multiple seconds. Also you need verify correlation across multiple Kafka topics.

A note for myself as an opensource pet project.

The project should allow users to:

• Consume on-demand Kafka topics from configuration
• Provide free-text-search on every field of Kafka message
• Search should be fast < 500ms with 50~100 million messages
• Render Kafka message as json value
• Create page for complex search which requires correlation across multiple Kafka messages
• Self-cleanup data store with configuration retention length (in days)

The app:

The UI:

I have used the following stacks for above application, and it works pretty well:

• Vaadin 23
• Solr 8

The UI is quick and responsive to query, result grid is lazy loading. Hard disk load is manageable and it is self-cleaning as well.