April 22, 2024 • 15 min read
Nobody cares about observability until checkout is down on a Saturday. At The Good Guys we ran the online store on top of 100-odd physical stores, millions of transactions a day. A couple of seconds of extra latency on a product page, or a payment step quietly failing for one card type — that's real money, and you won't see it in a dashboard unless you went looking for it first. We ran Mixpanel, New Relic and Sumo Logic. They overlap less than the marketing suggests; each answers a question the others can't. Black Friday was where I found out whether the setup actually held — traffic was somewhere north of 50x a normal day.
Detection went from 15 minutes to 30 seconds, checkout failures from 2.3% to 0.4%. At our volume those two numbers are most of why this paid for itself.
Mixpanel was where we went to understand behaviour. It's event-based rather than page-based, so instead of pageviews you record the things that matter — viewed a product, added to cart, completed a purchase — and build funnels out of them. That made it straightforward to see exactly where people dropped out of checkout. We tracked the events with their relevant properties:
// Mixpanel e-commerce tracking implementation
import mixpanel from 'mixpanel-browser';
class EcommerceAnalytics {
// Track critical e-commerce events
trackProductView(product) {
mixpanel.track('Product Viewed', {
product_id: product.id,
product_name: product.name,
category: product.category,
price: product.price,
in_stock: product.stock > 0,
page_load_time: performance.now(),
viewport_size: {
width: window.innerWidth,
height: window.innerHeight
}
});
}
trackPurchase(order) {
mixpanel.track('Purchase Completed', {
order_id: order.id,
revenue: order.total,
items: order.items.map(item => ({
product_id: item.product_id,
quantity: item.quantity,
price: item.price
})),
payment_method: order.payment_method,
first_time_customer: this.isFirstTimeCustomer()
});
// Track revenue
mixpanel.people.track_charge(order.total);
}
}Mixpanel tells you the customer dropped at payment. New Relic tells you it was a 4-second call to the payment gateway, on the third Tuesday, when their API was having a bad day. It traced the .NET app from request down through the database queries and outbound calls. We added custom instrumentation on the payment path specifically, because that's the bit where a slow span turns into a refund:
// New Relic .NET custom instrumentation
using NewRelic.Api.Agent;
[Transaction]
public class CheckoutService
{
[Trace]
public async Task<PaymentResult> ProcessPayment(PaymentRequest request)
{
NewRelic.SetTransactionName("Ecommerce", "ProcessPayment");
NewRelic.AddCustomAttribute("payment_method", request.PaymentMethod);
NewRelic.AddCustomAttribute("order_value", request.Amount);
try
{
var result = await paymentGateway.ProcessAsync(request);
NewRelic.RecordMetric("Custom/Payment/Success", 1);
return result;
}
catch (PaymentException ex)
{
NewRelic.RecordMetric("Custom/Payment/Failure", 1);
NewRelic.AddCustomAttribute("payment_failure_reason", ex.Reason);
throw;
}
}
}The point isn't that one is better. They sit in a chain. Sumo Logic was where all the logs landed, so once New Relic pointed at a slow service we could go read the actual lines for that request. The order I'd hit them in:
# The Good Guys E-commerce Observability ROI (2019-2021) Investment: $240,000 annually - Mixpanel Pro: $48,000/year - New Relic Enterprise: $120,000/year - Sumo Logic Professional: $60,000/year - PagerDuty + integrations: $12,000/year Measurable Business Returns: ┌─────────────────────────┬─────────────┬────────────────┬─────────────────┐ │ Metric │ Before │ After │ Annual Impact │ ├─────────────────────────┼─────────────┼────────────────┼─────────────────┤ │ Checkout Failure Rate │ 2.3% │ 0.4% │ +$2.1M revenue │ │ Mean Incident Detection │ 15 minutes │ 30 seconds │ +$950K revenue │ │ Page Load Time (P95) │ 4.2s │ 1.8s │ +18% conversion │ │ Uptime During Peak │ 99.2% │ 99.97% │ +$1.2M revenue │ └─────────────────────────┴─────────────┴────────────────┴─────────────────┘ Total Annual ROI: $4.25M revenue protection + performance gains ROI Multiple: 17.7x return on observability investment
The licences weren't cheap, and it's fair to ask whether three tools is one too many. For us it wasn't — each answered a different question, and the overlap was small. The honest version of the lesson is that observability only earns its keep if you wire the alerts to the things that cost money and then actually act on them. Collecting the data is the easy part.