A detailed analysis of ServiceNow production instance outages

Executive Summary

ServiceNow outages can stem from planned or unplanned issues at multiple layers. Planned maintenance/upgrades render instances briefly unavailable (e.g. “instance is within the maintenance window” message^{【21†L110-L114】【47†L149-L153】}). Cloud or data center outages (e.g. AWS/Azure region failures) can knock out connectivity; AWS even reported DNS and internal load-balancer failures causing global service disruption^{【56†L203-L211】}. Network routing problems (BGP route leaks or ISP peering issues) can isolate regions – for example a recent BGP leak blocked Australian users from US services^{【73†L195-L203】}. DNS or CDN edge failures can cause partial reachability. Load balancer or WAF/security blocks (misconfigured IP filters or DDoS mitigation) may throttle or blackhole traffic from some regions. ServiceNow-specific limits (API rate limits or semaphore queues) will reject excess requests with HTTP 429^{【66†L89-L93】}. Authentication/SSO failures (IdP downtime or misconfiguration) block logins. Database/mid-tier problems or instance cluster issues can degrade or halt the platform; admins can inspect stats.do and sys_cluster_state to see node and database status^{【71†L46-L54】【36†L218-L221】}. Config errors or custom code (infinite loops, broken business rules) may saturate the instance. External integrations or VPN/MPLS issues can cut off certain user groups. In rare cases hardware faults, instance data corruption, license/account suspension, or legal restrictions could cause outages.

We present a detailed scenario-by-scenario breakdown, including diagnostic steps (e.g. ping, dig, traceroute, BGP looking glasses, cloud status APIs, ServiceNow stats.do/threads.do/syslogs, Now Support portal) and remediation strategies (failovers, rollback to snapshot, contacting support), along with expected symptoms and region-specific behavior. A summary table compares likelihood, impact, detection, and remediation for each scenario.

ServiceNow Platform-Level Issues

• Scheduled Maintenance / Upgrades (Platform Maintenance Window): ServiceNow regularly patches or upgrades instances. During the maintenance window the instance shows a notice (“instance is unavailable…within maintenance window”)^{【21†L110-L114】【47†L149-L153】}. Root cause: intentional downtime for database or app-server patching. Diagnostics: Check the Now Support Maintenance Center (via support portal) or on-instance status page; watch for email notifications of upgrade schedule. (instance.service-now.com/esc?id=service_status on some portals). On the instance, the System Diagnostics > Stats page (stats.do) will show an F5 load-balancer offline and no active sessions^{【71†L46-L54】}. Symptoms: All users see a maintenance message or 503. Access is blocked globally (both admins and end-users). Some regions may intermittently appear “up” if replication is lagging or if cached, but essentially none can work during the window. Region-specific: Maintenance is not regionally segmented – it affects all. However, time zone timing matters: an upgrade in AEST may momentarily favor Asian access until US/EU business hours. Remediation: Wait out the window or rollback (restore pre-upgrade snapshot if catastrophic). If unscheduled prolonged outage occurs, contact ServiceNow Support immediately. Preventive: Plan upgrades in off-hours, stagger by region; configure maintenance schedules via CMDB/Now Support; subscribe to release notifications.

• Unplanned ServiceNow Platform Outage: Rarely, issues in the ServiceNow multi-tenant platform could cause a widespread “platform down”. These are usually announced on the official ServiceNow Status Page or via support channels. Root cause: internal cloud failures or software bugs. Diagnostics: Check the official status site (or crowd-sourced monitors like 503Radar) and ServiceNow community channels. On-instance, admins see the same outage message. Symptoms: All instances or multiple across tenants may report problems. Remediation: Engage ServiceNow Support. Preventive: None on customer side; rely on ServiceNow’s resilience. (Note: personal dev instances have no SLA; production instances do).

• Instance-Level Failures (Software Crash or Node Fault): An application server or critical service (e.g. mid-tier) could crash or hang. Root cause: JVM crash, out-of-memory, hung threads, etc. Diagnostics: Try to log in as sys_admin – if UI unresponsive, check ServiceNow’s System Diagnostics > Stats (stats.do) and Threads (threads.do) if accessible. Look at System Logs (syslog table) and System Health Monitor in the Instance Troubleshooter (if enabled). Check sys_cluster_state table to verify all app nodes and DB servers are online; for example hostnames like app128039.mel201.service-now.com indicate Melbourne nodes^{【36†L218-L221】}. Commands/Tools: Use stats.do to see if active sessions and queue depth are maxed, and threads.do for stuck transactions. Look for recent stack traces in system log. Symptoms: Instance may partially work (e.g. homepage loads) but key services fail. Or it may be completely hung/frozen. Region-specific: A software fault on an Australian node affects all users equally; no region works unless SN had a geo-redundant failover (generally not automatic). Remediation: Identify the failing node via sys_cluster_state; if one node crashed, SN may restart it or remove from load. Rolling back any recent configurations or code changes may be needed. Preventive: Monitor node health with the System Diagnostics dashboards; proactively update Java heap settings, and avoid heavy synchronous loads.

Infrastructure and Cloud Issues

• Data Center or Region Outage: If the entire Melbourne data center (or paired AZ) hosting the instance suffers a power/network failure, the instance is unreachable. Similarly, SN has mirrored secondary DCs but failover isn’t automatic. Root cause: physical infrastructure failure (power, network, fire suppression), or catastrophic event. Diagnostics: No access to instance – check ServiceNow’s Data Center Status portal or contact support. Use cloud provider status APIs (AWS Health, Azure Service Health) for region issues. Tools: AWS CLI/Azure CLI to query health events, or open cloud status pages. Symptoms: All users (APMEA/US/EU) see outage. Possibly a brief alternate path (if any exist) could allow some read-only access. Region-specific: None – all regions see it down. However, if SN has a nearby DR (rare for customers), Asia-Pacific users might see different latency. Remediation: ServiceNow might manually failover to the secondary DC, but this takes hours. (As explained by ServiceNow support, migrating an instance to another region is possible but requires a restoration and downtime of up to ~1 hour^{【36†L144-L153】}.) Preventive: Customers should back up critical work (update sets/export) and consider multi-instance setups. Monitor SN incident announcements and subscribe to support alerts.

• Cloud Provider (AWS/Azure/GCP) Regional Issues: Outages at the underlying hyperscaler (e.g. AWS Sydney, Azure Southeast Asia) will take SN instances down. For example, an AWS DNS misconfiguration and internal load-balancer fault led to global service disruptions^{【56†L203-L211】}. Root cause: Cloud network or service failure (routing, DNS, LB). Diagnostics: Check cloud status pages (e.g. AWS Health Dashboard, Azure status, Google Cloud Status). Use dig/nslookup on the instance’s DNS name to see if it resolves; use traceroute from various locations (e.g. EU vs APAC) to find where packets drop. Tools like BGP looking glasses or RIPE Atlas probes can identify regional blackholes. Symptoms: If the provider’s issue is regional, users in that region (e.g. US East) can’t reach the instance even if it’s on AWS Sydney; users near the data center (APMEA) might still connect. In one AWS event, DNS failures prevented access to DynamoDB API^{【56†L203-L211】}, illustrating how a cloud-level DNS issue could block the platform. Region-specific: If AWS US-East goes down, an SN instance on AWS AP-Southeast (Sydney) should not be directly affected, but network path anomalies (BGP routing, i.e. if US routes leak) could cause global routing chaos. Conversely, a Sydney region outage would cut off all global traffic to that instance, but maybe local (APAC) connections drop last. Remediation: Use cloud failover (if architected), or wait for cloud fix. Ensure your SN instance’s cloud account has multi-AZ or backup location. Preventive: For mission-critical, consider SN’s new Azure migration options (e.g. Microsoft moved its SN workload to Azure^{【54†L108-L116】}) or cross-region replication. Monitor cloud provider RSS and APIs for early warning.

Network Routing and Edge Services

• BGP Routing/Peering Issues: A bad BGP announcement or peering failure can isolate parts of the Internet. For example, a May 2023 route leak in the Angola Cables network caused Australian users to be unable to reach many US services (AWS, Akamai)^{【73†L195-L203】}. Root cause: Misconfigured route advertisement or transit provider issue. Diagnostics: If only certain regions are affected, run traceroute from affected and unaffected regions to the instance. Use public looking-glass/BGP tools (e.g. Hurricane Electric BGP Toolkit, RIPEstat) to check current AS paths and route announcements for your instance’s IP prefixes. A sudden “hole” or very long path indicates a leak or hijack. Check known interconnect outages. Symptoms: If a BGP leak directs traffic into a dead path, users in that zone see timeouts, while others still reach the instance. E.g., Australian and nearby users might still connect (since local routes are fine), but Europe/US lose connectivity^{【73†L195-L203】}. Remediation: Often the leak self-heals or network engineers withdraw the offending announcement. You may route around by using an alternate Internet egress (SD-WAN) or VPN through a different country. For critical services, multi-homing or using multiple DNS providers can help. Preventive: Monitor path health (RouteViews), use RPKI/ROA where possible, and keep alternate ISP links ready.

• DNS Resolution Problems: If DNS servers fail to resolve the instance name, users cannot find it. Root cause: DNS server misconfiguration, caching issues, or propagation delay. Diagnostics: Run dig instance.service-now.com from multiple locations (APAC, EU, US) and compare. Check TTLs and SOA. Use dig +trace to see each delegation. Check if your internal DNS or ISP DNS has stale records. On the SN side, ensure instance’s DNS entries are correct in the SN cloud network. Symptoms: Some users see “server not found”, others may still connect if their DNS has a cached entry. This can cause apparent region-specific availability if only certain DNS resolvers (e.g. local ISP in Europe) are failing while others are fine. Remediation: Flush DNS caches (ipconfig /flushdns or restart DNS service), switch to alternate DNS (8.8.8.8 / 1.1.1.1). Have secondary CNAME records. Preventive: Use global DNS and low TTL, monitor DNS health (services like DNSViz).

• CDN / Edge Cache Issues: If ServiceNow delivers some content (images, attachments, scripts) via a CDN or edge network (e.g. Akamai, Cloudflare), an edge outage could slow or break parts of the UI. Root cause: CDN node failure or routing to edge. Diagnostics: Use browser developer tools: if the page loads but static assets 404 or hang (e.g. attachments icons missing), it may be a CDN outage. Ping or curl the CDN domain. Symptoms: The portal UI may display incomplete pages or time out loading. This often affects only certain regions if an edge POP is down. Remediation: Bypass CDN by accessing instance directly (some SN admins can toggle debug to disable static acceleration). Wait for CDN fix. Preventive: None on user side; SN may multi-CDN their assets.

• Load Balancer / F5 Failures: ServiceNow front-ends use an F5 load balancer (or cloud LB) to spread traffic among app nodes. If the LB fails or misconfigures, the instance is unreachable. Root cause: LB software bug, SSL misconfiguration, full connection table. Diagnostics: If you have CLI or network access to the DMZ, ping or curl the LB IP. In SN, stats.do shows the F5 status^{【71†L46-L54】}. If LB is “offline” or showing errors, that’s the problem. Symptoms: All users fail to connect even though back-end nodes might be up. Region-specific: Unlikely to vary by region since LB is global. But internal vs external paths may differ (e.g. if LB VIP is region-locked). Remediation: Only SN support can fix their LB. They may failover to a redundant LB if configured. Preventive: SN regularly updates LB firmware; monitor health. Customers should ensure any corporate firewalls allow LB IP ranges.

• Firewall/WAF or Geofencing Blocks: A web/application firewall (external or on the ServiceNow network) might block traffic based on IP or geolocation. ServiceNow itself does not expose a user-configurable WAF (it’s handled internally), but firewalls (e.g. corporate) could. Root cause: A malicious attack causing SN or network to block entire regions (e.g. blocking all EU IPs to mitigate DDoS), or simple ACL misconfiguration. Diagnostics: Try accessing from a different network (3G, a VPN). On the corporate side, examine firewall logs for blocked connections to SN. Check if your IP was accidentally blacklisted by SN. Symptoms: Users in blocked zones get no response or a generic error; others connect normally. Remediation: Update firewall/WAF rules to allow SN’s IP range or domain. If SN has blocked you (e.g. via Abuse lists), contact SN Support. Preventive: Maintain an allowlist of SN IPs (published in SN docs) and monitor for auto-block triggers.

• DDoS Attacks and Mitigation: A Distributed Denial of Service (DDoS) attack can saturate the instance or its network, making it unusable. ServiceNow likely has DDoS protections (their Trust portal discusses their security posture^{【60†L105-L112】}). Root cause: Malicious traffic flood. Diagnostics: Observe an unusual surge in inbound connections on the stats page or firewall. ISP or cloud may flag DDoS. Symptoms: Complete outage for normal users; attackers’ traffic may or may not get a response. Some ISPs/regions may route around the attack, causing uneven impact. Remediation: If under DDoS, SN or cloud will implement scrubbing/filtering. They may also geofilter (blocks) offending traffic; this could incidentally drop legitimate traffic. Preventive: Use SN’s Security Operations for incident detection, ensure any custom WAF rules are tuned, and consider multi-cloud failover.

• Rate Limiting / Throttling: ServiceNow enforces API rate limits and semaphores. If integrations or users flood the instance, further calls are rejected with HTTP 429 (“Too Many Requests”)^{【66†L89-L93】}. Root cause: Exceeding ServiceNow’s defined throughput (governance limits or semaphore queue). Diagnostics: Look for HTTP 429 errors in the browser/app. Use stats.do to see if active sessions are at semaphore capacity. Symptoms: Slow performance or error pages for high-volume operations; the instance is still technically up (other operations not hitting limits may work). Typically this affects automated integrations or APIs rather than UI, but a runaway script could stall interactive logins. Region-specific: Throttling is per-instance, so should not vary by region. Remediation: Spread API calls across multiple integration users (each has its own rate limit) as recommended by SN. Increase capacity if on an Enterprise plan. Preventive: Design integrations to use asynchronous/batch APIs, back off on 429, and monitor usage with the Stats Tools app.

• Authentication/SSO/Identity Provider Failures: If ServiceNow is configured with SAML/OAuth SSO, an IdP outage or certificate expiry will block logins. Root cause: IdP (e.g. Azure AD, Okta) downtime, expired SSO cert/key, or misconfigured SAML settings. Diagnostics: Attempt login via the native ServiceNow login (if enabled) or a different IdP. Check IdP status dashboards. Inspect SSO logs in ServiceNow (Single Sign-On > SAML > SAML Logs) or IdP audit logs. Symptoms: Users get SSO-specific errors (e.g. “Assertion Invalid” or “Provider not responding”). The instance itself is healthy – only auth is broken. Region-specific: If corporate users rely on a regionally-hosted IdP (e.g. onsite ADFS), those in that region may fail while others (with global IdP) work. Remediation: Roll back to simple password login (disable SAML) if urgent, then fix IdP (renew cert, re-establish trust). Preventive: Configure fallback admin login; monitor IdP certificates; ensure multiple sign-on methods.

Backend and Application Layer Failures

• Database or Mid-Tier Performance Degradation: If the SN database or mid-tier services slow (e.g. due to locks, resource exhaustion), the instance may become sluggish or unresponsive. Root cause: Long-running queries, full transaction logs, or deadlocks. Diagnostics: In Stats (stats.do), watch for abnormally high average response times and queue depths. Use the Database Locks list in SN or reach out to Now Support to inspect DB health. Symptoms: Pages time out or error out intermittently; background jobs backlog. Region-specific: No regional split – all users hit the same DB. Remediation: Kill rogue queries (SN support). Possibly restore from a recent clone or re-index large tables. Preventive: Use indexes, avoid very large exports in peak times, and upgrade instance tier if near capacity.

• Instance Cluster / Replication Failures: ServiceNow runs clusters of app servers and mirrored DB nodes. If replication between nodes fails, data inconsistency or downtime can occur. Root cause: Network glitch between nodes, or node crash leading to split-brain. Diagnostics: Check sys_cluster_state: it lists each app and DB node and their status^{【36†L218-L221】}. If a node is “offline”, that component is unreachable. SN control nodes or logs will show replication errors. Symptoms: If a primary DB loses its standby, failover might not occur on crash, risking data loss. If an app server is out, that node simply won’t serve UI. Region-specific: Only relevant to the instance’s own cluster (not multi-region). Remediation: Contact SN support to repair cluster (rejoin node, rebuild replication). Preventive: Keep instance on current release (ensures cluster stability), and use SN’s High Availability offerings where available.

• Tenant Isolation / “Noisy Neighbor”: In multitenant clouds, a noisy neighbor may impact performance. Root cause: Another tenant’s faulty instance causing underlying hypervisor issues. Diagnostics/Symptoms: Rare for ServiceNow (they isolate very strongly). It would manifest as unexplained slowness across nodes. Mitigation/Preventive: Contact SN support – they can migrate your instance to a different hardware pool.

Configuration, Customizations, and Integrations

• Configuration Errors: Mistakes in security rules, HTTP proxies, or IP ACLs could block traffic. Example: A corporate proxy that disallows *.service-now.com domains would prevent login. Or network ACL mistakenly blocking SN IP blocks. Diagnostics: Review recent changes. Test access from an unfiltered network. Symptoms: “Connection refused” or blank pages without obvious errors. Remediation: Revert the bad rule, or whitelist the instance. Preventive: Document config changes and keep backups of network device configs.

• Custom Code / Business Rules: Flawed business rules or scripts (e.g. infinite loops, expensive GlideRecord queries) can hang the instance or exhaust resources. Root cause: Poorly written logic triggered on common transactions. Diagnostics: Temporarily disable recent custom business rules; check System Logs for script engine exceptions. Use System Diagnostics > Stats to see if a particular transaction dominates CPU (look at “longest transactions”). Symptoms: Instance slows to a crawl; administrators might see “Session timed out” errors. Remediation: Disable the faulty rule and test. Roll back code update sets. Preventive: Always test scripts in sub-prod; implement best practices (e.g. avoid synchronous loops, add logs to detect hangs).

• Third-Party Integrations / External Dependencies: External web services (e.g. Jira, AWS, etc.) failing could cascade. Example: A synchronous mid-tier integration that waits on an unavailable service could tie up threads. Diagnostics: Check System Logs for external call timeouts. Use External Web Service logs or MID server logs. Symptoms: Specific functionality breaks, and possibly UI hangs on that action. But the core platform remains up. Region-specific: If integration servers are region-specific, only some users see errors. Remediation: Disable or delay the integration, switch to asynchronous patterns. Preventive: Use circuit-breaker patterns and fallback logic.

Network and Connectivity

• Corporate VPN / SD-WAN / MPLS Issues: If business users connect through a corporate tunnel, a failure in that network will make the instance unreachable to them, even if Internet users are fine. Diagnostics: Have affected users try a direct Internet connection or VPN exit in another region. Check VPN device logs. Symptoms: Users on one network see timeout; users elsewhere can use ServiceNow normally. Remediation: Fix the corporate link or reroute through alternate POPs. Preventive: Use redundant links or allow split-tunnel to fallback to public Internet.

• ISP / Regional Peering Problems: Outages or congestion at an ISP can isolate certain users. Diagnostics: Check Downdetector for reports from affected countries; use traceroute to see where the path fails. Tools like BGP looking glasses can show where the transit path is cut. Symptoms: One region’s users time out, others are unaffected. Remediation: Encourage users to switch ISP or use mobile data; wait for ISP resolution. Preventive: Have multiple ISPs or cellular backup for critical staff.

• Time Zone / Unannounced Maintenance: Sometimes automated maintenance (backups, scans) is run in regional midnight hours and inadvertently impacts performance. Diagnostics: Correlate the timeline of outage with known maintenance windows. (SN typically schedules outside business hours but on your local night.) Check syslog around that time for any scheduled job spikes. Remediation: If coincidental maintenance is detected, reschedule it. Preventive: Coordinate all cross-regional jobs to avoid simultaneous impact.

Rare and Administrative Causes

• Hardware Faults / Corrupted Data: Very rare, but e.g. a failed disk in the SAN or RAM error could cause the instance to crash or lose data consistency. Diagnostics: Only ServiceNow’s internal monitoring (they manage hardware) would detect this. Symptom might be repeated crashes or “internal error” messages. Remediation: ServiceNow will restore from the last good backup and rebuild any failed hardware. Preventive: Trust ServiceNow’s redundancy; make regular clone snapshots.

• Licensing/Account Suspension: If a customer’s contract lapses or violates terms, ServiceNow could suspend the instance. Diagnostics/Symptoms: Possibly a message on login saying “Account on hold” or no message but no access. This is unlikely without notice. Remediation: Contact sales/support to resolve any billing or license issues. Preventive: Keep subscription active and in compliance.

• Legal/Regulatory Blocks: In some rare jurisdictions (e.g. data sovereignty laws), access might be restricted. Example: If a country government blocks the ServiceNow domain. Diagnostics: Check local ISPs or regulatory announcements. Symptoms: Some countries cannot reach the service, others can (e.g. China blocking Google, similarly could block SN). Remediation: None short of legal appeal or using a proxy. Preventive: Choose a compliant data center and legal counsel for international operations.

Diagnostic Tools & Commands

• Ping/Traceroute: ping <instance>.service-now.com to test basic reachability. traceroute <instance> from different continents to locate routing failures.
• DNS Lookup: dig +trace <instance>.service-now.com or nslookup to verify DNS resolution path.
• curl/HTTP Tests: From a shell, curl -v https://<instance>.service-now.com to inspect HTTP status codes (429, 503, etc).
• ServiceNow Stats Pages: Log in as admin and go to System Diagnostics > Stats (stats.do) to view real-time performance, active sessions, and load balancer status^{【71†L46-L54】}. The Threads page shows blocked or long-running transactions.
• Logs and System Diagnostics: In ServiceNow, check System Logs (e.g. syslog_transaction) and Instance Troubleshooter if installed. Also examine the sys_cluster_state table to see each node’s health and physical server name^{【36†L218-L221】}.
• Cloud Status APIs: Use AWS CLI (aws health describe-events), Azure CLI (az monitor activity-log list), or GCP status page API for region incidents.
• BGP Looking Glass: Use public tools (e.g. curl https://bgp.he.net/AS20940) to verify how other networks see routes to SN. The RIPEstat RIS API can show route announcements to the IP.
• Monitoring/Alerting: Ensure SN’s System Events and Jobs Dashboard (System Diagnostics) is used. Configure synthetic transactions (e.g. external monitors pinging the instance from multiple geolocations). Leverage SN’s Instance Troubleshooter or third-party uptime monitors.

Summary Comparison Table

Network Diagram (Example)

flowchart LR
	subgraph Users
	  U1[User (APMEA Region)] 
	  U2[User (EU/US Region)]
	end
	subgraph Internet
	  DNS["DNS Resolver"]
	  LB[Load Balancer (F5/Cloud LB)]
	end
	subgraph ServiceNow_Instance
	  App1[App Server Node 1 (melbourne)]
	  App2[App Server Node 2 (melbourne)]
	  DB[(Database Cluster)]
	end
	U1 -- Internet --> DNS
	U2 -- Internet --> DNS
	DNS --> LB
	LB --> App1
	LB --> App2
	App1 --> DB
	App2 --> DB

This flow illustrates how users from any region reach the ServiceNow instance via global DNS and a load balancer to app servers and the database. Region-specific issues (e.g. BGP problems) might block the path from U2 (EU/US) to DNS or LB, explaining why APMEA (U1) could still connect when others cannot.

Failover Path Diagram (Hypothetical)

flowchart LR
	A[Primary DC (Melbourne)] -->|Replication| B[Secondary DC (Sydney)]
	A --(if failed)--> B
	A -- Normal --> Users[ServiceNow Users]
	B -- Failover --> Users

If ServiceNow were to failover the instance to a secondary region (with manual intervention, as explained by SN support^{【36†L144-L153】}), traffic would be rerouted from the Primary DC (Melbourne) to the Secondary DC. This ensures continuity but requires planned downtime to sync data.

Conclusion

ServiceNow instance outages can arise from many layers. By methodically checking maintenance schedules, status pages, network paths, DNS, logs, and ServiceNow diagnostics tools, administrators can pinpoint the root cause. The above scenarios cover almost every plausible failure mode—from routine upgrades to exotic BGP leaks. Quick detection (via monitoring/alerts) and having rollback or workaround plans (like alternate login or backup networks) can greatly reduce downtime. Regular best practices (proactive monitoring, patching, and multi-region planning) will prevent or mitigate most issues.