Ingesting Financial Tick Data Using a Time-Series Database

·

Financial tick data—granular, high-frequency records of every trade and quote in a market—is essential for algorithmic trading, risk modeling, and real-time analytics. As data volumes grow exponentially, traditional databases struggle with ingestion speed and query performance. This is where time-series databases (TSDBs) like QuestDB come into play.

QuestDB is a high-performance, open-source time-series database optimized for handling financial market data. With native support for InfluxDB Line Protocol (ILP), fast SQL analytics, and efficient storage architecture, it's ideal for ingesting and analyzing tick-level data from crypto exchanges and traditional financial markets.

In this guide, we’ll explore three robust methods to ingest crypto market data into QuestDB for real-time analysis:

We'll walk through setup steps, code examples, and best practices to ensure seamless integration.

Prerequisites

Before diving in, you need a running instance of QuestDB. Start by creating a local environment:

mkdir cryptofeed-questdb
cd cryptofeed-questdb
docker run \
  -p 9000:9000 -p 9009:9009 -p 8812:8812 -p 9003:9003 \
  -v "$(pwd)":/var/lib/questdb \
  questdb/questdb:8.1.0

This command launches QuestDB with key ports exposed:

Once running, access the web interface at http://localhost:9000 to monitor incoming data.

Method 1: Ingest Data Using the Cryptofeed Library

For developers seeking rapid integration, Cryptofeed is one of the most effective tools. This Python library connects via WebSocket to major exchanges—including Binance, OKX, Kraken, and Gemini—and normalizes data into a consistent format.

👉 Discover how to build a real-time crypto data pipeline using open-source tools and high-speed ingestion.

Why Use Cryptofeed?

Step-by-Step Setup

Create a virtual environment and install Cryptofeed:

python3 -m venv cryptofeed
source cryptofeed/bin/activate
pip install cryptofeed

Next, create questdb.py with the following code to stream BTC-USDT trades from OKX and Gemini:

from cryptofeed import FeedHandler
from cryptofeed.backends.quest import TradeQuest
from cryptofeed.defines import TRADES
from cryptofeed.exchanges import OKX, Gemini

QUEST_HOST = '127.0.0.1'
QUEST_PORT = 9009

def main():
    f = FeedHandler()
    f.add_feed(OKX(channels=[TRADES], symbols=['BTC-USDT'], callbacks={TRADES: TradeQuest(host=QUEST_HOST, port=QUEST_PORT)}))
    f.add_feed(Gemini(channels=[TRADES], symbols=['BTC-USDT'], callbacks={TRADES: TradeQuest(host=QUEST_HOST, port=QUEST_PORT)}))
    f.run()

if __name__ == '__main__':
    main()

Run the script: python questdb.py. Data will begin flowing into QuestDB automatically.

In the web console, query your data:

SELECT * FROM "trades-OKX" LIMIT 10;
💡 Note: Table names follow the pattern "trades-{exchange}". You may not see immediate results from Gemini due to lower trade frequency.

Customizing Ingestion with Callbacks

Need control over table names or column structure? Override the default callback:

class TradeQuest(QuestCallback, BackendCallback):
    default_key = 'trades'

    async def write(self, data):
        update = f'{self.key},symbol={data["symbol"]},side={data["side"]} price={data["price"]},amount={data["amount"]} {int(data["timestamp"] * 1_000_000_000)}'
        await self.queue.put(update)

This approach lets you define custom ILP messages—perfect for aligning with downstream analytics systems.

Core Keywords: tick data, time-series database, market data ingestion, crypto trading, real-time analytics, QuestDB, Influx Line Protocol, WebSocket streaming

Method 2: Build a Custom Market Data Pipeline

When Cryptofeed doesn’t support your target exchange—or you need preprocessing logic—a custom pipeline gives full control.

Here’s an example using the QuestDB Python SDK to pull price data from Binance and Gemini via REST APIs and push it via ILP:

import requests
import time
from questdb.ingress import Sender, TimestampNanos

conf = 'http::addr=localhost:9000;'

def get_binance_data():
    url = 'https://api.binance.us/api/v3/ticker/price'
    params = {'symbol': 'BTCUSDT'}
    response = requests.get(url, params=params)
    data = response.json()
    btc_price = float(data['price'])
    print(f"BTC Price on Binance: {btc_price}")
    publish_to_questdb('Binance', btc_price)

def get_gemini_data():
    url = 'https://api.gemini.com/v1/pubticker/btcusdt'
    response = requests.get(url)
    data = response.json()
    btc_price = float(data['last'])
    print(f"BTC Price on Gemini: {btc_price}")
    publish_to_questdb('Gemini', btc_price)

def publish_to_questdb(exchange, price):
    print("Publishing BTC price to QuestDB...")
    with Sender.from_conf(conf) as sender:
        sender.row(
            'prices',
            symbols={'pair': 'BTCUSDT'},
            columns={'exchange': exchange, 'bid': price},
            at=TimestampNanos.now()
        )
        sender.flush()

def job():
    print("Fetching BTC price...")
    get_binance_data()
    get_gemini_data()

while True:
    job()
    time.sleep(5)

This script polls every 5 seconds and writes to a prices table. Use this pattern for unsupported exchanges or when applying filters, aggregations, or anomaly detection before storage.

👉 Learn how low-latency data pipelines empower advanced trading strategies.

Method 3: Ingest Market Data Using Change Data Capture (CDC)

For enterprise environments, Change Data Capture (CDC) minimizes polling overhead by streaming only changed records from source systems like Kafka or PostgreSQL.

Imagine a scenario where a data team publishes crypto prices to a Kafka topic. Instead of polling APIs directly, you use the QuestDB Kafka Connector to listen for changes and stream them in real time.

CDC Architecture Overview

  1. A service polls Coinbase or other sources.
  2. New prices are pushed to a Kafka topic.
  3. The QuestDB Kafka Connector consumes messages and inserts them into QuestDB.

This decoupled design improves scalability and fault tolerance.

For implementation details, refer to the official guide: Realtime Crypto Tracker with QuestDB Kafka Connector.

Frequently Asked Questions (FAQs)

What is Level 1, Level 2, and Level 3 market data?
Level 1 includes last price, best bid/ask. Level 2 adds market depth (top 5–10 orders). Level 3 provides full order book visibility with individual trader actions—common in equities markets.

What is an order book?
An order book lists all buy (bids) and sell (asks) orders for an asset, organized by price level. It updates continuously as trades occur or orders are modified.

What are candlestick charts used for?
Candlestick charts display open, high, low, and close prices over time. They help traders identify trends, reversals, and volatility patterns.

How does InfluxDB Line Protocol work?
ILP is a text-based format for writing time-series data: table,tags values timestamp. QuestDB accepts ILP over TCP for high-throughput ingestion.

Is QuestDB suitable for high-frequency trading (HFT)?
Yes. With sub-millisecond ingestion latency and SQL analytics optimized for time-series workloads, QuestDB supports HFT backtesting and real-time monitoring.

Can I use PostgreSQL clients with QuestDB?
Absolutely. QuestDB supports the PostgreSQL wire protocol on port 8812, enabling compatibility with existing BI tools like Tableau or Metabase.

Final Thoughts

Ingesting financial tick data efficiently requires a database built for speed and scale. QuestDB excels in this domain with fast ingestion via ILP, powerful SQL queries, and flexible connectivity options.

Whether you’re using Cryptofeed for quick setup, building a custom pipeline for control, or deploying CDC for enterprise integration, QuestDB adapts to your workflow.

As real-time decision-making becomes critical across finance and fintech, mastering these ingestion patterns ensures you stay ahead of the curve.

👉 Explore how modern time-series databases are transforming financial data analysis.