Skip to content

rv-live

Stream BGP updates from RouteViews via Kafka.

Synopsis

bgpipe [...] -- rv-live [OPTIONS]

Description

The rv-live stage consumes real-time BGP updates from the University of Oregon RouteViews project via its public Kafka stream. RouteViews collects BGP data from routers at major Internet exchange points and transit providers worldwide.

The stream carries BGP messages in OpenBMP raw format. The stage parses BMP messages, extracts the inner BGP messages, and injects them into the pipeline tagged with PEER_IP, PEER_AS, COLLECTOR (the RouteViews BGP collector, eg. linx) and rv-live/collector (the internal BMP machine, eg. bmp1), eg. for filters like tag[COLLECTOR] == linx.

By default, the stage subscribes to all topics matching the pattern ^routeviews\..+\.bmp_raw$, which covers all RouteViews collectors. Use --collector to select specific collectors by name prefix, or --topics to supply a custom topic regex.

RouteViews injects the collector's own AS into the AS_PATH of each message. By default, this stage strips that first hop. Use --keep-aspath to preserve the original AS_PATH as received from the collector.

The --state option enables offset persistence: the stage saves its Kafka consumer position to a file and resumes from the same point on restart. This prevents re-processing messages after a pipeline restart.

Over long distances, each broker sends only about 1 MB/s per connection, which is not enough for the whole RouteViews feed. The stage therefore reads with --consumers parallel Kafka consumers in one consumer group. Messages from one topic stay in order, but messages from different topics can interleave differently between runs.

Connection retries are enabled by default. The stage monitors data freshness and reconnects when no data arrives for --stale duration.

The stream carries one prefix per UPDATE. To put the original UPDATEs back together, add combine: bgpipe -o -- rv-live -- combine.

Different RouteViews peers may use 2-byte or 4-byte ASNs, and some send ADD-PATH Path Identifiers in their NLRI. bgpipe detects both automatically (see the global --as4 and --addpath options).

Options

Option Type Default Description
--broker string stream.routeviews.org:9092 Kafka broker address
--topics string ^routeviews\..+\.bmp_raw$ Topic regex pattern
--collector strings Only subscribe to collectors whose name starts with this prefix
--collector-not strings Exclude collectors whose name starts with this prefix
--group string (auto-generated) Kafka consumer group ID
--state string State file for offset persistence
--refresh duration 5m Topic list refresh interval
--timeout duration 30s Connection timeout
--stale duration 3m Reconnect if no data for this long; 0 disables
--consumers int 4 Number of parallel Kafka consumers
--retry bool true Retry connection on errors
--retry-max int 0 Max retry attempts; 0 means unlimited
--keep-aspath bool false Keep the collector AS in AS_PATH

Examples

Stream all RouteViews updates to stdout:

bgpipe -o -- rv-live

Stream from a specific collector:

bgpipe -o -- rv-live --collector linx

Exclude specific collectors:

bgpipe -o -- rv-live --collector-not amsix --collector-not decix

Persist consumer state for resumable processing:

bgpipe -o -- rv-live --state /var/lib/bgpipe/rv-live.state

Archive RouteViews data with RPKI validation:

bgpipe \
    -- rv-live \
    -- rov --invalid keep \
    -- write --every 1h 'rv-updates.$TIME.json.gz'

See Also

ris-live, combine, read, RouteViews, Stages overview