rv-live¶
Stream BGP updates from RouteViews via Kafka.
Synopsis¶
Description¶
The rv-live stage consumes real-time BGP updates from the University of Oregon RouteViews project via its public Kafka stream. RouteViews collects BGP data from routers at major Internet exchange points and transit providers worldwide.
The stream carries BGP messages in OpenBMP raw format. The stage parses
BMP messages, extracts the inner BGP messages, and injects them into the
pipeline tagged with PEER_IP, PEER_AS, COLLECTOR (the
RouteViews BGP collector, eg. linx) and rv-live/collector (the internal BMP
machine, eg. bmp1), eg. for filters like tag[COLLECTOR] == linx.
By default, the stage subscribes to all topics matching the pattern
^routeviews\..+\.bmp_raw$, which covers all RouteViews collectors.
Use --collector to select specific collectors by name prefix, or
--topics to supply a custom topic regex.
RouteViews injects the collector's own AS into the AS_PATH of each
message. By default, this stage strips that first hop. Use --keep-aspath
to preserve the original AS_PATH as received from the collector.
The --state option enables offset persistence: the stage saves its Kafka
consumer position to a file and resumes from the same point on restart.
This prevents re-processing messages after a pipeline restart.
Over long distances, each broker sends only about 1 MB/s per connection, which is
not enough for the whole RouteViews feed. The stage therefore reads with --consumers
parallel Kafka consumers in one consumer group. Messages from one topic stay in order,
but messages from different topics can interleave differently between runs.
Connection retries are enabled by default. The stage monitors data
freshness and reconnects when no data arrives for --stale duration.
The stream carries one prefix per UPDATE. To put the original UPDATEs back
together, add combine: bgpipe -o -- rv-live -- combine.
Different RouteViews peers may use 2-byte or 4-byte ASNs, and some send ADD-PATH
Path Identifiers in their NLRI. bgpipe detects both automatically (see the global
--as4 and --addpath options).
Options¶
| Option | Type | Default | Description |
|---|---|---|---|
--broker |
string | stream.routeviews.org:9092 |
Kafka broker address |
--topics |
string | ^routeviews\..+\.bmp_raw$ |
Topic regex pattern |
--collector |
strings | Only subscribe to collectors whose name starts with this prefix | |
--collector-not |
strings | Exclude collectors whose name starts with this prefix | |
--group |
string | (auto-generated) | Kafka consumer group ID |
--state |
string | State file for offset persistence | |
--refresh |
duration | 5m |
Topic list refresh interval |
--timeout |
duration | 30s |
Connection timeout |
--stale |
duration | 3m |
Reconnect if no data for this long; 0 disables |
--consumers |
int | 4 |
Number of parallel Kafka consumers |
--retry |
bool | true |
Retry connection on errors |
--retry-max |
int | 0 |
Max retry attempts; 0 means unlimited |
--keep-aspath |
bool | false |
Keep the collector AS in AS_PATH |
Examples¶
Stream all RouteViews updates to stdout:
Stream from a specific collector:
Exclude specific collectors:
Persist consumer state for resumable processing:
Archive RouteViews data with RPKI validation: