Skip to content

CrocSwap/dex-market-analytics

BranchesTags

Repository files navigation

dex-market-analytics

Research tools for collecting and analyzing data across perp dexs and cexs

Latest Updates:

  • ✅ Fixed Binance order book crossing issues (snapshot-based updates)
  • ✅ Fixed Lighter market index mapping and incremental updates
  • ✅ Added automatic gz compression for capture files
  • ✅ Grid extractor uses client timestamps by default (avoids clock skew)
  • ✅ PYTH_LAZER symbols normalized (BTC/USD → BTC)

Overview

This library provides a standardized, type-safe interface for connecting to various perpetual DEX and CEX WebSocket APIs, as well as ultra-low latency price oracle feeds. It abstracts away the differences between exchanges while providing a unified data model for market data consumption.

Features

  • 🔄 Real-time WebSocket connections with automatic reconnection
  • 📊 Standardized data types across all DEX and CEX sources
  • 🛡️ Full TypeScript support with strict typing
  • 🏗️ Extensible architecture for adding new exchanges
  • 💪 Robust error handling and connection management
  • 📈 Support for order books and trades streaming
  • ⚡ Ultra-low latency price feeds via Pyth Lazer
  • 📦 Data capture system with automatic compression
  • 📊 Grid extraction for time-series analysis
  • 🔄 Multi-exchange capture with unified format

Quick Start

Installation

yarn install

Basic Usage

Hyperliquid

import { HyperliquidWebSocketClient, MarketDataHandler, Exchange, MarketDataChannel } from 'dex-market-analytics';

// Define your handlers
const handlers: MarketDataHandler = {
  onTrade: (trade) => console.log('New trade:', trade),
  onOrderBookSnapshot: (snapshot) => console.log('Order book update:', snapshot),
  onConnect: () => console.log('Connected'),
  onError: (error) => console.error('Error:', error)
};

// Create and connect client
const client = new HyperliquidWebSocketClient(handlers);
client.connect();

// Subscribe to market data
client.subscribe({
  exchange: Exchange.HYPERLIQUID,
  symbol: 'BTC',
  channels: [MarketDataChannel.TRADES, MarketDataChannel.ORDERBOOK]
});

Lighter

import { LighterWebSocketClient, MarketDataHandler, Exchange, MarketDataChannel } from 'dex-market-analytics';

// Define your handlers
const handlers: MarketDataHandler = {
  onTrade: (trade) => console.log('New trade:', trade),
  onOrderBookSnapshot: (snapshot) => console.log('Order book update:', snapshot),
  onConnect: () => console.log('Connected'),
  onError: (error) => console.error('Error:', error)
};

// Create and connect client
const client = new LighterWebSocketClient(handlers);
client.connect();

// Subscribe to market data
client.subscribe({
  exchange: Exchange.LIGHTER,
  symbol: 'BTC',
  channels: [MarketDataChannel.TRADES, MarketDataChannel.ORDERBOOK]
});

EdgeX

import { EdgeXWebSocketClient, MarketDataHandler, Exchange, MarketDataChannel } from 'dex-market-analytics';

// Define your handlers
const handlers: MarketDataHandler = {
  onTrade: (trade) => console.log('New trade:', trade),
  onOrderBookSnapshot: (snapshot) => console.log('Order book update:', snapshot),
  onConnect: () => console.log('Connected'),
  onError: (error) => console.error('Error:', error)
};

// Create and connect client (public WebSocket)
const client = new EdgeXWebSocketClient(handlers);
client.connect();

// Subscribe to market data
client.subscribe({
  exchange: Exchange.EDGEX,
  symbol: 'BTC',
  channels: [MarketDataChannel.TRADES, MarketDataChannel.ORDERBOOK]
});

Running Examples

# DEX Examples
yarn example:hyperliquid   # Hyperliquid WebSocket
yarn example:lighter       # Lighter WebSocket
yarn example:edgex         # EdgeX WebSocket
yarn example:aster         # Aster WebSocket
yarn example:pacifica      # Pacifica WebSocket
yarn example:paradex       # Paradex WebSocket
yarn example:apex          # ApeX WebSocket

# CEX Examples
yarn example:binance       # Binance WebSocket
yarn example:bybit         # Bybit WebSocket

# Price Oracle Examples
yarn example:pyth          # Pyth SSE (~180ms latency)
yarn example:pyth-ws       # Pyth Lazer WebSocket (sub-ms latency)

Pyth Lazer Setup (Ultra-Low Latency)

For Pyth Lazer WebSocket with sub-millisecond latency:

# Set your authentication token
export PYTH_LAZER_TOKEN=your-token-here
yarn example:pyth-ws

# Or create a .env file (see .env.example)
cp .env.example .env
# Edit .env with your token
yarn example:pyth-ws

Architecture

Standardized Types

The library uses standardized types that work across all supported DEXs:

Core Market Data Types

  • Trade: Standardized trade data

    • exchange: Source exchange identifier
    • symbol: Trading pair/symbol
    • price: Executed price
    • size: Trade size
    • side: BUY or SELL
    • timestamp: Unix timestamp
    • tradeId: Unique trade identifier

  • OrderBookSnapshot: Full order book state

    • bids: Array of bid levels (price, size, orderCount)
    • asks: Array of ask levels (price, size, orderCount)
    • timestamp: Snapshot timestamp
    • symbol: Trading pair
    • exchange: Source exchange

  • OrderBookLevel: Individual price level

    • price: Price level
    • size: Total size at this level
    • orderCount: Number of orders (optional)

Subscription Management

  • MarketDataSubscription: Define what data to receive

    • exchange: Target exchange
    • symbol: Trading symbol
    • channels: Array of data channels (TRADES, ORDERBOOK)

  • MarketDataHandler: Event callbacks

    • onTrade: Handle incoming trades
    • onOrderBookSnapshot: Handle order book updates
    • onConnect: Connection established
    • onDisconnect: Connection lost
    • onError: Error occurred

Base Classes

BaseWebSocketClient

Abstract base class providing:

  • Automatic reconnection with exponential backoff
  • Ping/pong heartbeat management
  • Connection state management
  • Event-driven architecture
  • Subscription tracking

Supported Exchanges

Decentralized Exchanges (DEXs)

Hyperliquid (HyperliquidWebSocketClient)

  • Mainnet and testnet support
  • L2 order book streaming
  • Real-time trade data
  • Automatic subscription management

Lighter (LighterWebSocketClient)

  • Mainnet WebSocket support
  • Order book incremental updates (not snapshots)
  • Real-time trade data
  • Market index-based subscription system (ETH=0, BTC=1)
  • Optional authentication for account-specific channels
  • Note: Market indices don't follow typical ordering

EdgeX (EdgeXWebSocketClient)

  • Public and private WebSocket endpoints
  • Order book depth levels (15 or 200 levels)
  • Real-time trade streams with liquidation tracking
  • Ticker data with funding rates
  • Contract metadata support
  • Automatic ping-pong handling

Aster (AsterWebSocketClient)

  • Binance-style WebSocket API
  • Aggregate trade streams
  • Order book depth updates
  • 24hr ticker statistics
  • Combined and raw stream modes
  • Up to 1001x leverage support

Pacifica (PacificaWebSocketClient)

  • Hyperliquid fork with compatible API
  • Order book and trade streaming
  • Automatic reconnection support

ApeX (ApexWebSocketClient) ⚠️

  • STATUS: Not Functional - API Issues
  • ApeX Omni DEX WebSocket implementation
  • Issue: WebSocket returns "handler not found" errors for all documented channel formats
  • Possible causes:
    • May require authentication even for public data
    • Documentation may be outdated or incorrect
    • API endpoint may be non-functional
  • Note: Disabled by default in capture config until API issues are resolved
  • Symbol format: BTC-USDT

Paradex (ParadexWebSocketClient) ⚠️

  • STATUS: Not Functional - Requires Authentication
  • StarkNet-based perpetual DEX
  • Issue: Requires bearer token authentication before subscribing
  • Order book updates via delta messages
  • Trade execution streaming
  • Market statistics and funding rates
  • Note: Disabled by default in capture config due to auth requirements

Centralized Exchanges (CEXs)

Binance (BinanceWebSocketClient)

  • Industry-standard WebSocket API
  • Aggregate trades and order book depth
  • Mini ticker and 24hr ticker data
  • Mark price and funding rate streams
  • Multi-stream support

Bybit (BybitWebSocketClient)

  • V5 API WebSocket support
  • Order book depth (1, 50, 200, 500 levels)
  • Public trade streaming
  • Ticker data with funding info
  • Kline/candlestick data
  • Liquidation tracking

Price Oracles

Pyth Network (PythLazerClient)

  • SSE Implementation (~180ms latency)
    • Server-Sent Events via Hermes API
    • Price feed streaming with confidence intervals
    • Automatic symbol resolution

Pyth Lazer (PythLazerWebSocketClient)

  • WebSocket Implementation (sub-millisecond latency)
    • Direct WebSocket connection for ultra-low latency
    • Requires authentication token
    • High-frequency price updates
    • Failover endpoint support
    • Environment variable configuration for secure token management

Development

Project Structure

dex-market-analytics/
├── src/
│   ├── types/
│   │   ├── market-data.ts          # Standardized types
│   │   ├── hyperliquid.ts          # Hyperliquid-specific types
│   │   ├── lighter.ts              # Lighter-specific types
│   │   ├── edgex.ts                # EdgeX-specific types
│   │   ├── aster.ts                # Aster-specific types
│   │   ├── apex.ts                 # ApeX-specific types
│   │   ├── binance.ts              # Binance-specific types
│   │   ├── bybit.ts                # Bybit-specific types
│   │   ├── paradex.ts              # Paradex-specific types
│   │   ├── pyth-lazer.ts           # Pyth SSE types
│   │   └── pyth-lazer-ws.ts        # Pyth WebSocket types
│   ├── clients/
│   │   ├── base-websocket-client.ts         # Base WebSocket abstraction
│   │   ├── hyperliquid-websocket-client.ts  # Hyperliquid implementation
│   │   ├── lighter-websocket-client.ts      # Lighter implementation
│   │   ├── edgex-websocket-client.ts        # EdgeX implementation
│   │   ├── aster-websocket-client.ts        # Aster implementation
│   │   ├── pacifica-websocket-client.ts     # Pacifica implementation
│   │   ├── paradex-websocket-client.ts      # Paradex implementation
│   │   ├── apex-websocket-client.ts         # ApeX implementation
│   │   ├── binance-websocket-client.ts      # Binance implementation
│   │   ├── bybit-websocket-client.ts        # Bybit implementation
│   │   ├── pyth-lazer-client.ts             # Pyth SSE client
│   │   └── pyth-lazer-websocket-client.ts   # Pyth WebSocket client
│   ├── examples/
│   │   └── [exchange]-example.ts    # Usage examples for each exchange
│   └── index.ts                      # Main exports
├── dist/                             # Compiled JavaScript (generated)
├── .env.example                      # Environment variables template
├── .gitignore                        # Git ignore configuration
├── tsconfig.json                     # TypeScript configuration
└── package.json                      # Dependencies and scripts

Available Scripts

  • yarn dev - Development mode with auto-reload
  • yarn build - Compile TypeScript to JavaScript
  • yarn start - Run compiled application
  • yarn type-check - Type-check without compiling
  • yarn clean - Remove build artifacts
  • yarn capture - Run multi-exchange data capture
  • yarn grid - Extract time-series grid from captures

Example Scripts

  • yarn example:hyperliquid - Hyperliquid DEX streaming
  • yarn example:lighter - Lighter DEX streaming
  • yarn example:edgex - EdgeX DEX streaming
  • yarn example:aster - Aster DEX streaming
  • yarn example:pacifica - Pacifica DEX streaming
  • yarn example:paradex - Paradex DEX streaming
  • yarn example:apex - ApeX DEX streaming
  • yarn example:binance - Binance CEX streaming
  • yarn example:bybit - Bybit CEX streaming
  • yarn example:pyth - Pyth SSE price feeds
  • yarn example:pyth-ws - Pyth Lazer WebSocket (ultra-low latency)

Adding New Exchanges

To add support for a new DEX:

  1. Create exchange-specific types in src/types/{exchange}.ts
  2. Extend BaseWebSocketClient in src/clients/{exchange}-websocket-client.ts
  3. Map exchange data to standardized types
  4. Add to Exchange enum in src/types/market-data.ts

Example structure:

export class NewExchangeClient extends BaseWebSocketClient {
  protected handleMessage(data: WebSocket.Data): void {
    // Parse exchange-specific message
    // Convert to standardized types
    // Call appropriate handlers
  }

  protected resubscribe(): void {
    // Resubscribe after reconnection
  }
}

Type Reference

See Type Documentation for detailed type definitions and usage.

Data Capture System

Multi-Exchange Capture

# Run capture with default configuration
yarn capture

# Environment variables
export PYTH_LAZER_TOKEN=your-token  # Required for Pyth Lazer

Features:

  • Captures data from all configured exchanges simultaneously
  • Automatic file rotation (hourly by default)
  • Automatic gz compression of completed files (~90% space savings)
  • State management for crash recovery
  • Real-time metrics and monitoring

Grid Extraction

# Extract 1-second grid from captures
yarn grid data/captures/

# Custom period and symbol filter
yarn grid -p 5000 -s BTC data/captures/

# Use exchange timestamps instead of client time
yarn grid --exchange-time data/captures/

Features:

  • Streaming processing for large datasets
  • Handles both .jsonl and .jsonl.gz files
  • Client timestamps by default (avoids clock skew)
  • Symbol normalization (BTC/USD → BTC)
  • CSV output with bid/ask/mid/spread

Known Issues

Exchange-Specific Issues

Binance

  • Fixed: Order book crossing issue (was treating snapshots as incremental)

Lighter

  • Fixed: Market index mapping (ETH=0, BTC=1, not the reverse)
  • Fixed: Incremental updates were treated as snapshots

ApeX

  • Non-functional: API returns "handler not found" for all subscriptions
  • Disabled by default in capture config

Paradex

  • Requires authentication: Even for public data
  • Disabled by default in capture config

PYTH_LAZER

  • Symbols include /USD suffix (normalized in grid extraction)
  • Requires authentication token

License

MIT

About

Research tools for collecting and analyzing data across perp dexs and cexs

Resources

Readme
Activity
Custom properties

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages