Synthetic Monitoring for SaaS: How to Independently Verify SLAs

Key takeaways:

Your enterprise information technology (IT) leadership has likely considered migrating some on-premise services to a public cloud or replacing them with software-as-a-service (SaaS) products.

Common migration targets include product lifecycle management (PLM) capabilities, enterprise resource planning (ERP) systems, customer relationship management (CRM) solutions, and human resource information systems (HRIS).

Migrations can impair digital user experiences in very tangible ways. Your employees or customers may see long response times, frequent failures during routine tasks, and unpredictable unavailability of functions.

Though felt by users, such impacts may be intangible or even invisible in vendor observability metrics. It often feels like external vendors take on infrastructure ownership without proportionate business accountability.

Find out how you can ensure business accountability from your vendors through synthetic monitoring of SaaS applications using Keysight's Eggplant Monitoring.

Why does traditional monitoring face visibility failures with SaaS and cloud?

Figure 1. Traditional on-premise deployment and monitoring

Most traditional web application monitoring and application performance monitoring (APM) tools are designed for on-premise environments. They assume that all layers of the hardware and software stack will report metrics with high granularity, enabling deep insights into the entire stack.

Figure 2. SaaS deployments are invisible to traditional monitoring

But when applications move outside the corporate firewall to a public cloud or SaaS, this fundamental assumption is broken. Traditional monitoring cannot bypass firewall barriers unless the vendor permits them inside.

Information barriers are worse. If SaaS providers don't provide deep metrics of their internal workings, there's nothing for traditional monitoring to measure. As a result, during and after cloud migrations and SaaS adoptions, traditional tools often go blind with massive visibility gaps.

The six R's framework outlines common enterprise strategies for digital transformation, cloud migration, and SaaS adoption. Four of them are relevant to us here.

Rehosting and replatforming

You lift and shift most or all of your on-premise components to a public cloud's infrastructure-as-a-service (IaaS).

Sometimes, a few components, like databases, may be replatformed to cloud platform-as-a-service (PaaS) solutions if the latter have very similar interfaces. However, a provider's IaaS/PaaS cloud monitoring tools don't provide as extensive and granular metrics as your on-premise IT infrastructure monitoring.

Repurchasing

This drop and shop strategy is more common for high-level applications such as PLM or CRM systems. You drop your on-premise application entirely and migrate all capabilities to a fully managed SaaS solution. Existing essential customizations are re-architected for the chosen SaaS.

Most SaaS vendors don't provide deep observability into their services. Only high-level application uptime and availability status are provided. Common problems like latency spikes and random failures for individual customers are neither visible nor documented by the vendor.

Re-architecting

This strategy rewrites a custom complex application into a cloud-native application that uses microservices with third-party PaaS/SaaS dependencies. It combines replatforming and repurchasing, inheriting the visibility gaps of both.

What are common accountability gaps in cloud migration and SaaS adoption?

These technical visibility gaps amplify into major business-level accountability gaps and adverse outcomes as outlined below.

Inadequacy of service level agreements (SLAs)

SLAs are an essential legal tool to enforce accountability on vendors. However, they're far from ideal.

For example, a large public cloud's SLAs only talk about a low-granularity uptime metric and only offer credit refunds for their breakage. You can hold them accountable only if you're able to technically break down your issues as emerging from the vendor's downtime and prove it.

No client-specific metrics or evidence

Vendors provide only low-granularity average metrics across all their customers, such as average uptime and average error rates. Individual customer latencies or failures are not voluntarily reported or documented by vendors, even if severe.

No accountability due to information asymmetry

When any client-specific performance issues occur, the client must technically prove that the root cause is not internal but lies on the vendor's side.

Vendors, on the one hand, don't provide access to your monitoring. At the same time, they also demand technical evidence. This fundamental asymmetry and unfairness lead to unproductive mutual blame cycles without actually resolving the underlying problem.

No real-time insights

Most IT issues start as minor delays or failures. If unaddressed early on, they tend to snowball into major disruptions over time.

That's why your IT teams love real-time alerting.

However, this key requirement is broken in SaaS deployments. Early detection, if any, only happens within the vendor's infrastructure. If they fail to report it in real time to your IT teams, the opportunity to quickly mitigate a business disruption is lost.

Absence of user perspectives

A key deficiency is that vendor SLAs don't measure what's relevant to your users or their user actions. They don't do digital experience monitoring or user performance monitoring as felt by your customers or employees.

Vendors calculate uptime differently from how your users actually experience performance. Latencies during an essential end-to-end workflow may be a major issue for your employees, but the vendor's SLA may only measure uptimes.

No independent verification

Availability dashboards are completely under the control of vendors. Reported values are not independently verified. There are no industry-level bodies or regulators to standardize or verify their data.

Inadequate compensation for disruptions

Most vendor SLAs compensate for issues with refunds or credits. However, their compensation may not match the actual intangible long-term harm due to lost business, damaged reputation, or employee frustration.

Inconsistent user experiences

SaaS vendors don't deliver consistent end-user experiences to all geographies. Your US employees' experiences of a business-critical SaaS may be very different from those in Singapore. These can lead to productivity bottlenecks.

What are good solutions for enterprise SaaS visibility and accountability?

Based on the above problems, a good monitoring approach for SaaS and cloud must have the following characteristics:

Two common monitoring approaches that can implement these characteristics are:

How does synthetic monitoring compare with RUM?

Synthetic monitoring collects metrics by periodically simulating realistic user interactions with a SaaS using scripted automation, while RUM collects metrics during actual user interactions. Both synthetic monitoring and RUM provide insights into different aspects of SaaS interactions.

The benefits of synthetic monitoring are outlined below:

The benefits of RUM are listed here:

The above comparisons show that the two monitoring systems are not alternatives but are complementary.

Why should you choose Keysight Eggplant Monitoring for synthetic monitoring?

Keysight Eggplant Monitoring is Keysight's powerful synthetic monitoring service. Its key capabilities are outlined below.

Fully managed by Keysight

A key differentiator is that Keysight Eggplant Monitoring is a fully managed service from Keysight. No installations or other overheads are required. User journey scripts are written and maintained by Keysight experts.

At the same time, being fully managed doesn't mean it’s inflexible. It integrates with your existing IT infrastructure, including public and hybrid clouds. Customizations for your use cases, script editing, and integrations with your preferred services are possible.

Monitoring of user journeys, web applications, and web pages

Figure 3. Monitors for user journey and web pages in Keysight Eggplant Monitoring

Keysight Eggplant Monitoring can perform user-perspective testing by automating complex, multi-step user journeys. Various aspects like end-to-end monitoring, transaction monitoring, and middleware API monitoring are covered. It can test across diverse technology layers, including:

It enables comprehensive views through high-granularity metrics for every step in a user journey, providing crucial validation. It can also monitor for website performance optimization.

Independent SLA verification

Keysight Eggplant Monitoring does all its automated testing from independent datacenters. Your issues can be replicated from independent locations and network connections.

All observations are accompanied by auditable evidence, such as timestamps and screenshots. You can back your vendor conversations with irrefutable data and evidence from fully independent monitoring.

Client-side monitoring

By using a real Chrome browser for its automated tests, Keysight Eggplant Monitoring can capture client-side metrics like page load times and first contentful paint to identify web performance and usability issues.

False positives prevention

If there are failures, each is double-checked via a retest to minimize false positives. This ensures that most alerts are triggered only by genuine issues.

Continuous assurance throughout SaaS and cloud migrations

Keysight Eggplant Monitoring enables business continuity monitoring throughout your enterprise's digital transformation, as outlined below:

For example, Lloyd's Register uses Eggplant Monitoring to ensure optimal user experiences across their services for the maritime industry.

Real-time alerting

Figure 4. Real-time monitoring dashboard

For every enabled monitor, active monitoring is executed on a regular schedule with a frequency determined by your teams. For example, they can configure higher frequencies for SLA adherence tests and lower frequencies for less critical applications. The service sends alerts as soon as issues are detected, often based on predefined thresholds, long before your users start calling IT. Your teams can configure multiple levels of escalation contacts. Each contact can be alerted through SMS, voice, email, or third-party services (such as PagerDuty, Slack, and ServiceNow).

Mobile app access

The Keysight Eggplant Monitoring mobile apps enable quick visibility for your IT teams supporting global operations.

API access

The Keysight Eggplant Monitoring API enables integration of real-time monitoring with internal management dashboards for SLA compliance, outage detection, workflow failures, and performance trends.

It also enables your IT teams to integrate real-time monitoring with DevOps.

Insightful visualizations and reports

Figure 5. Keysight Eggplant Monitoring reports

Keysight Eggplant Reporting has an extensive set of visualizations and reporting features to facilitate troubleshooting and performance optimizations.

The images below show various visualizations and charts available in the monitoring portal.

Figure 6. Results of user journey monitoring

Figure 7. Granular performance metrics from a user journey step

Figure 8. Performance distribution of test durations for a monitor

Rely on Keysight Eggplant for synthetic monitoring of SaaS applications

SaaS vendors provide services without proportionate accountability for business outcomes. Synthetic monitoring tools help to reduce this accountability gap by giving you visibility into end-to-end performance across SaaS boundaries.

Keysight Eggplant Monitoring is a powerful synthetic monitoring solution that facilitates independent assurance for your digital transformation, cloud migrations, and SaaS adoptions.

Contact us to see how Keysight Eggplant Monitoring works and request a demonstration.

Related Posts

limit
3