beta
Borderfree Unscheduled Downtime
Incident Report for PB Products
Postmortem

POST-INCIDENT ANALYSIS: Borderfree Downtime 07/30/2019

Dear Pitney Bowes Merchant,

Your customers may have experienced brief intermittent interruptions to Borderfree's Envoy and Checkout API services on Tuesday, Jul 30, 2019. Our checkout service remained available throughout the entire timeframe, but successful orders would have been intermittent for some of our customers.

Our technical team has completed their investigation of this issue, as summarized below.

Root cause

From our investigation, we have determined that one of the nodes from our backend high availability (HA) checkout pool was failing to processes approximately 50% of the requests. Our system health monitors showed that the underlying servers were operating normally, which prolonged our ability to diagnose and remedy the issue. The culprit was an anomaly in the application on only one of the HA nodes. Once we were able to pinpoint the offending node, the service was recycled and our checkout service returned to processing 100% of incoming orders.

Corrective actions

We are performing a comprehensive review to identify the cause of the abnormal behavior of the offending node. Additionally, we are reviewing our existing monitoring systems.

Incident details

Start time: 30 Jul 2019 18:25 UTC
End time: 30 Jul 2019 19:25 UTC
Duration: 60 minutes

We apologize for the inconvenience and thank you for your continued partnership with Pitney Bowes. If you would like more details on the impact of this issue, please email support@borderfree.com.

Regards,
The Borderfree Production Support Team

Posted 3 months ago. Aug 02, 2019 - 10:46 EDT

Resolved
Dear Pitney Bowes Merchant,

The issue that was preventing your international customers from checking out using our Envoy/API Checkout has been resolved.

All Borderfree services are online and fully operational.

We apologize for any inconvenience and thank you for your continued partnership with Pitney Bowes.

Regards,
The Borderfree Production Support Team
Posted 3 months ago. Jul 30, 2019 - 15:50 EDT
Update
Dear Pitney Bowes Merchant,

We are still experiencing an issue that may prevent your international customers from checking out using Envoy/API Checkout.

Our technical team is working to resolve this issue as quickly as possible.

If you have any questions, please contact support@borderfree.com.

Regards,
The Borderfree Production Support Team
Posted 3 months ago. Jul 30, 2019 - 15:21 EDT
Investigating
Dear Pitney Bowes Merchant,

As of 2:45 PM ET, we’ve been experiencing an issue that may prevent your international customers from checking out using Envoy/API Checkout.

Our team is actively investigating the issue. We will send an update as soon as possible.

If you have any questions, please contact support@borderfree.com.

Regards,
The Borderfree Production Support Team
Posted 3 months ago. Jul 30, 2019 - 14:53 EDT
This incident affected: Cross-Border Retail (Checkout - Envoy, Checkout - API).