Circuit Breakers for Flaky OSINT Sources: Fail Fast, Recover Quietly

A polite scraper hits a flaky source and waits 30 seconds for a TCP timeout. It does this 100 times an hour. That is 50 minutes of wall-clock time per hour spent waiting on one dead source. Meanwhile every other adapter is healthy and the dashboard looks fine. The fix is a circuit breaker, and in Elixir it is about 80 lines of GenServer wrapped around a few counters.

#The three states

A circuit breaker is a tiny state machine:

• closed — traffic flows, failures are counted.
• open — traffic is rejected immediately with {:error, :circuit_open}.
• half-open — one probe request is allowed; success closes, failure re-opens.

The trick is the transition rules. Too sensitive and you flap. Too tolerant and you are back to 50 minutes per hour of timeouts.

#The rules that actually work

• Open on: 5+ failures in a 30-second rolling window OR 3 consecutive failures.
• Cool-down: exponential, starting at 10s, doubling up to 5 minutes.
• Half-open probe: exactly one in-flight probe. Concurrent probes defeat the purpose.
• Close on: probe success.

defmodule PrismaticOsint.Breaker do
  use GenServer

  def call(name, fun) do
    case GenServer.call(name, :state) do
      :open   -> {:error, :circuit_open}
      :closed -> record(name, safely(fun))
      :half_open -> probe(name, fun)
    end
  end

  defp safely(fun) do
    try do
      {:ok, fun.()}
    catch
      kind, reason -> {:error, {kind, reason}}
    end
  end
end

The try/catch wraps the user function so a crash never leaves the breaker in an inconsistent state. The :error path feeds the counter; the :ok path resets it.

#Wire it into the pipeline

Every OSINT adapter gets wrapped at the boundary:

def search(query, opts) do
  Breaker.call(breaker_for(__MODULE__), fn ->
    HTTPoison.get!(url(query), [], recv_timeout: 5_000)
  end)
end

That is it. The adapter does not know about the breaker’s internal state. The pipeline does not know about the adapter’s flakiness. Both stay boring, which is the goal.

#Telemetry is non-negotiable

Every state transition emits telemetry:

:telemetry.execute([:osint, :breaker, :open], %{count: 1}, %{adapter: name})
:telemetry.execute([:osint, :breaker, :half_open], %{count: 1}, %{adapter: name})
:telemetry.execute([:osint, :breaker, :closed], %{count: 1}, %{adapter: name})

A breaker that opens and never closes is a source that died. A breaker that flaps is a source with intermittent issues (and possibly a wrong cool-down). Without observability you cannot tell which is which.

#Where to go next

• Academy: OTP Fundamentals — the GenServer under the breaker
• Glossary: Fault Tolerance, Retry, Rate Limiting, Telemetry, Observability

Fail fast. Recover quietly. Let the healthy sources do the work.