The Independent Observer Problem

The standard advice for monitoring is to keep at least one checker running somewhere independent of the thing it watches. Everything fails eventually, including the thing that tells you about failures.

The advice is right, but it hides a distinction. "Elsewhere" and "independent" are two different claims. A checker on another host can still depend on the same DNS provider, the same identity provider or the same cloud region as the site it watches. Move it to another server and you have changed its address, not necessarily its fate.

Push the idea further and it never settles. Any independent observer is also software, on a host, behind a domain, and any of those can go down. Add a second observer to watch the first and you have the same problem one level up. Absolute control is not on offer.

What is on offer is more useful, and it is what this post is about.

The chain never ends, so stop climbing it

The obvious reaction to "the monitor can fail" is to monitor the monitor. Then you notice the second monitor can fail too, and the honest end of that line of thought is an infinite stack of watchers, each one supposedly more reliable than the last.

Nobody builds that, and nobody should. The goal is not an observer that cannot fail. The goal is fewer blind spots shared between the observer and the observed. Two systems that fail for different reasons are worth more than ten that fail for the same one.

That turns a philosophical problem into an inventory problem, and inventories are something you can actually finish.

Draw three boxes

Independence is a property of dependencies, so write them down. For most setups there are three things to map:

1. The site. Where it is hosted, which DNS provider answers for it, which CDN sits in front, who issued the certificate, and which third parties it needs to work at all: login, payments, search.
2. The checker. Where the monitoring service runs, which DNS resolvers it uses, which network and cloud provider it sits on, and how you log in to it.
3. The alert path. Everything between "the checker noticed" and "a human knows": the mail provider, the Slack workspace, the Telegram bot, the phone, the person holding it, and whether they are awake.

Then draw a line wherever two boxes share something. Each line is a failure that can take out the site and your knowledge of it at the same moment.

Some of them are obvious once written down:

• Same cloud region. A self-hosted checker on the same provider and region as production goes dark in exactly the regional outage you most need to hear about.
• Same DNS provider. If your DNS host has a bad hour, your site stops resolving, and so does a checker that resolves its own hostnames through the same provider.
• Same identity provider. If staff log in to the monitoring dashboard through the SSO that is also failing, you may get the alert and be unable to open it.
• Same mail stack. If alerts go out through the same mail provider your application uses for everything else, a mail outage silences both the product and the warning about it.
• Same person. If the only person who receives alerts is also the only person who can fix the outage, holidays are a dependency too.

None of these is exotic. They are the defaults you get when you set things up quickly, which is how most monitoring gets set up.

Elsewhere is not the same as independent

This distinction is worth testing separately.

Elsewhere is about location. Is the checker on a different machine, network or provider? You can answer that by looking at an IP address.

Independent is about cause. Is there any single failure that takes out both the site and the checker? You can only answer that by walking the dependency map, and the answer is usually "yes, a few".

A managed monitoring service is elsewhere by default, which is a large part of its value. It is not automatically independent. If your site and your monitor both sit behind the same CDN, or your alerts land in a Slack workspace that is down for the same reason your site is, the hosting bill being different does not help.

The alert path is the part nobody tests

Most teams check the checker at some point. Far fewer check the last mile.

A monitor that detects an outage and then fails to tell anyone is indistinguishable, from the outside, from no monitor at all. And alert paths decay quietly: a Slack app gets uninstalled during a workspace clean-up, a bot token is rotated, the on-call person changes teams and their email starts bouncing, a mail filter decides the alerts are bulk mail.

None of that produces an error you will see during a normal week, because during a normal week nothing is down.

So test delivery on purpose, on a schedule, and write down when you did it. In Barkme every channel has a test button that sends a real message through the real integration. When Slack, Linear or Trello reject the stored credentials during a delivery, the channel is flagged on the channels page instead of failing in silence. Neither of those tells you a person read the message, so the test only counts once somebody confirms it arrived where they would have seen it at 3am. The alert nobody read goes deeper into routing.

Report the uncertainty instead of hiding it

The map has a second use. A monitoring setup should be able to describe its own limits. Not a green dashboard, but a short, honest report:

• What the setup can detect. The site returning errors, timing out, failing DNS or TLS, certificates and domains approaching expiry, the domain landing on a blocklist.
• What it cannot detect. A regional outage that also takes out the checker. A DNS provider failure that affects both. Content that is wrong but answers 200. A login flow that is broken behind a page that still loads.
• Which dependencies are still shared. The lines you drew between the three boxes, and the ones you chose to accept.
• When alert delivery was last tested, and by whom. A date, not a feeling.

Writing this down does not make the setup more reliable on its own. What it does is turn an invisible risk into a visible one. "We would not notice a failure at our DNS provider until a customer told us" is an uncomfortable sentence, and uncomfortable sentences are the ones that get fixed. A dashboard full of green ticks never prompts that conversation.

It also changes the promise. Nobody can promise absolute control over whether they will know about an outage. You can promise that you know which outages you would miss.

A practical version for a small team

You do not need a reliability programme for this. An afternoon is enough:

1. Draw the three boxes and list what each one depends on.
2. Mark every dependency that appears in more than one box.
3. For each shared one, decide: remove it, add a second path, or accept it and write it down.
4. Send a test alert through every channel and confirm a human saw it. Write down the date.
5. Put the short report somewhere your team will find it, and review it whenever you change hosting, DNS, mail or on-call.

The cheapest high-value move is usually the alert path, not the checker. A second delivery channel that shares nothing with the first, say a personal email next to the team Slack, removes more blind spots than a second monitoring service would.

The watcher can fail too

Barkme is a checker, so all of this applies to it as much as to anything else. It runs on servers, resolves names through DNS and delivers alerts through mail providers and third-party APIs, and any of those can have a bad day. If you use Barkme, it belongs in your second box with its dependencies written next to it, and your report should say so.

That is the honest version of independent monitoring. Not a chain of observers that never fails, but a small map of who depends on whom, a dated record of the last time the alarm actually rang, and a plain list of what you would still miss.

For what to check and how often, see how often you should check your site is up. For the failures that no request-level check will ever see, see the expiry dates that take sites down and how a domain ends up blacklisted.