Updates and maintenance
We deploy updates during market hours without stopping the service. Your clients may see a reconnection or a pause of a few seconds; they never lose data, as long as they follow the rules on this page.
The SDKs apply every rule on this page by themselves. Read on if you write your own client, or to know what your application will observe.
What happens during an update
| What is updated | Real time (WebSocket) | REST |
|---|---|---|
| The stream servers | A notice (kind: "reconnect"), then the connection is closed with code 1012 within 20 seconds. Reconnect and resume from your cursors: nothing is lost, nothing is sent twice. | — |
| The computation servers | The connection stays open. Each subscription receives a status source_delayed, then live once the computation has caught up, usually within 30 seconds. The delayed messages then arrive, in order. | — |
| How an indicator is computed, or its default values | Each subscription receives a reset (engine_change), a new subscribed, then a new snapshot. | A paginated query or an export started before the change cannot be continued: 410 CURSOR_EXPIRED. Start it again. |
| The API servers | — | Requests in progress complete. Nothing to do. |
A computation change is announced in advance with a maintenance notice. Among updates, it is the only one that requires rebuilding state: every other update is a pause followed by the missed messages.
Default values are set per ticker and regularly updated. An update of the defaults is an engine update: the engine of subscribed changes, every subscription receives a reset with engine_change, and a history query that omits those parameters uses the new defaults from then on, over past ranges too. A subscription that passes all its parameters explicitly is reset too, but its results do not change: to keep exactly the same computation over time, pass the parameters explicitly.
Maintenance notices
When we plan maintenance with a visible effect (a pause longer than usual, a computation change), every connection receives a notice, as soon as it is announced:
{"t":"notice","kind":"maintenance","id":"6f1c2a7e-0b5d-4c1e-9a3f-2d8e4b7c1a90","effectiveAt":"1791295200000000000","endsAt":"1791295500000000000","message":"Computation update: each subscription restarts from a new snapshot."}effectiveAtandendsAt: the window during which the service may pause, in nanoseconds since January 1, 1970 UTC, as strings. Show the delay to your users (effectiveAtminus now), or schedule your own jobs around it;- every new connection receives the notices of the windows not over yet, right after it opens: a client that connects after the announcement is warned too;
- a window can be moved: you then receive a notice with the same
idand new times. A cancellation is announced with{"t":"notice","kind":"maintenance-cancelled","id":"6f1c2a7e-0b5d-4c1e-9a3f-2d8e4b7c1a90"}; - the windows announced and not over are also listed by
GET /v1/status, inmaintenance, ordered by start time.
In GET /v1/status, each window has these fields. The start of the window is called startsAt there: it is the same instant as the effectiveAt of the notice.
| Field | Content |
|---|---|
id | The id of the window, the same as in the notices. |
startsAt | Start of the window, in nanoseconds since January 1, 1970 UTC, as a string. |
endsAt | End of the window, in the same format. |
message | The text of the announcement. |
{
"maintenance": [
{ "id": "6f1c2a7e-0b5d-4c1e-9a3f-2d8e4b7c1a90", "startsAt": "1791295200000000000", "endsAt": "1791295500000000000", "message": "Computation update: each subscription restarts from a new snapshot." }
]
}A notice is information: you have nothing to send back, and your subscriptions continue until the maintenance actually happens.
Rules for your WebSocket client
-
Keep, for each subscription, the last cursor and the
engineof its latestsubscribed. The cursor tells the server where you stopped;enginetells it which computation produced your state. -
Reconnect on
1012, on any close not listed as final on the WebSocket page, and after 15 seconds without any message, with a delay that doubles after each failure (up to 30 seconds) plus random jitter. -
Resume each subscription with both values:
JSON {"t":"subscribe","sub":"bt","instrument":"NQ.front","indicator":"big-trades","params":{"minimum":30},"mode":"confirmed","from":{"cursor":"20732.44294.1","engine":"82cbdecf93748dd5"}}If the computation changed while you were away, you receive
subscribed, areset(engine_change) and a snapshot, instead of messages computed differently from your state. Withoutengine, the server cannot tell, and you could keep objects that the new computation no longer produces. -
On a
reset, discard the state of the subscription and start again from the snapshot that follows. -
Treat
source_delayedas a pause, not an error. Keep the connection: the stream resumes by itself and sends what was late. -
Ignore any
upsertorremovewhose cursor is not after the last one you processed: after a reconnection, a message can arrive twice.
Rules for your REST client
- Retry a
503that carries aRetry-Afterheader after the delay it gives: the request was not processed. - Retry a
GETthat failed without a response (network error, connection closed) or with a502, after a short delay that grows with each attempt: a read has no effect other than its response. A query page or an export that failed without a response may have been computed and charged. Retrying it is harmless for your data, but the retry may be charged again. - Paginated queries: a
410CURSOR_EXPIREDon a next page means the computation changed between two pages. Start the query again from its first page, and drop the pages already read. - Exports: after an interruption, resume with the
X-Export-Tokenof the first response (see Exports). A410CURSOR_EXPIREDon the resume means the computation changed since the export started: start the export again, and drop the lines already received.
With the SDK
The SDK reconnects, resumes with the cursor and the engine of each subscription, drops duplicates, retries the REST requests the rules above allow, and resumes interrupted exports. Your code only sees events:
import { FathomChartsStream } from '@fathom-charts/sdk';
const stream = new FathomChartsStream({
url: 'wss://stream.fathomcharts.com/v1',
apiKey: process.env.FATHOM_CHARTS_API_KEY!,
onNotice: (notice) => {
if (notice.kind === 'maintenance') {
const inMinutes = Math.round((Number(BigInt(notice.effectiveAt!) / 1_000_000n) - Date.now()) / 60_000);
console.log(`Maintenance in ${inMinutes} min: ${notice.message}`);
}
},
});
for await (const event of stream.subscribe({ instrument: 'NQ.front', indicator: 'big-trades', params: { minimum: 30 } })) {
switch (event.type) {
case 'reset': /* discard the local state: a snapshot follows */ break;
case 'snapshot': /* the new starting state */ break;
case 'status': console.log('status', event.state); break; // source_delayed: a pause, then live
case 'notice': /* the same notices, in each subscription */ break;
}
}import asyncio
import os
import time
from fathom_charts import FathomChartsStream, Notice, Reset, Snapshot, Status
def on_notice(notice: Notice) -> None:
if notice.kind == "maintenance" and notice.effective_at is not None:
in_minutes = round((int(notice.effective_at) / 1e9 - time.time()) / 60)
print(f"Maintenance in {in_minutes} min: {notice.message}")
async def main() -> None:
async with FathomChartsStream(
url="wss://stream.fathomcharts.com/v1",
api_key=os.environ["FATHOM_CHARTS_API_KEY"],
on_notice=on_notice,
) as stream:
async with stream.subscribe(instrument="NQ.front", indicator="big-trades", params={"minimum": 30}) as sub:
async for event in sub:
match event:
case Reset():
pass # discard the local state: a snapshot follows
case Snapshot():
pass # the new starting state
case Status():
print("status", event.state) # source_delayed: a pause, then live
case Notice():
pass # the same notices, in each subscription
case _:
pass
asyncio.run(main())using FathomCharts;
await using var stream = new FathomChartsStream(new StreamOptions
{
Url = "wss://stream.fathomcharts.com/v1",
ApiKey = Environment.GetEnvironmentVariable("FATHOM_CHARTS_API_KEY"),
OnNotice = notice =>
{
if (notice.Kind == NoticeKind.Maintenance && notice.EffectiveAt is { } at)
{
var inMinutes = Math.Round((long.Parse(at) / 1_000_000 - DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()) / 60_000.0);
Console.WriteLine($"Maintenance in {inMinutes} min: {notice.Message}");
}
},
});
await foreach (var e in stream.Subscribe(new SubscribeOptions { Instrument = "NQ.front", Indicator = "big-trades", Params = new { minimum = 30 } }))
{
switch (e)
{
case Reset: /* discard the local state: a snapshot follows */ break;
case Snapshot<JsonValue>: /* the new starting state */ break;
case Status status: Console.WriteLine($"status {status.State}"); break; // SourceDelayed: a pause, then Live
case Notice: /* the same notices, in each subscription */ break;
}
}use std::time::{SystemTime, UNIX_EPOCH};
use fathom_charts::{FathomChartsStream, NoticeKind, StreamEvent, SubscribeOptions};
use futures::StreamExt;
use serde_json::json;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let stream = FathomChartsStream::builder("wss://stream.fathomcharts.com/v1")
.api_key(std::env::var("FATHOM_CHARTS_API_KEY").map_err(|_| "FATHOM_CHARTS_API_KEY is not set")?)
.on_notice(|notice| {
// Timestamps are nanoseconds: parse them as integers, never as f64 (above 2^53).
if let (NoticeKind::Maintenance, Some(Ok(at))) = (¬ice.kind, notice.effective_at.as_deref().map(str::parse::<u64>)) {
let now = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs();
let in_minutes = ((at / 1_000_000_000) as i64 - now as i64) / 60;
println!("Maintenance in {in_minutes} min: {}", notice.message.as_deref().unwrap_or(""));
}
})
.build()?;
let mut sub = stream.subscribe::<serde_json::Value>(
SubscribeOptions::new("NQ.front", "big-trades").params(json!({ "minimum": 30 })),
)?;
while let Some(event) = sub.next().await {
match event? {
StreamEvent::Reset(_) => { /* discard the local state: a snapshot follows */ }
StreamEvent::Snapshot(_) => { /* the new starting state */ }
StreamEvent::Status(status) => println!("status {}", status.state), // source_delayed: a pause, then live
StreamEvent::Notice(_) => { /* the same notices, in each subscription */ }
_ => {}
}
}
Ok(())
}stream.maintenancestream.maintenancestream.Maintenancestream.maintenance()lists the windows announced and not over;- to resume after your own process restarts, save the cursor and the engine of the subscription, then subscribe again from them (see Resume after a restart);
rest.exportNdjson()rest.export_ndjson()rest.ExportNdjsonAsync()rest.export_ndjson()resumes an interrupted export by itself; it throwsCURSOR_EXPIREDwhen the computation changed since the export started.