BigChange Ltd - Delays with Automations and Custom Actions – Incident details

All systems operational

Delays with Automations and Custom Actions

Resolved
Operational
Started almost 4 years agoLasted about 16 hours

Affected

BigChange

Operational from 2:54 PM to 7:05 AM

Client Site

Operational from 2:54 PM to 7:05 AM

Mobile Application Services

Operational from 2:54 PM to 7:05 AM

Updates
  • Resolved
    UTC
    Resolved

    The rollback was completed successfully. This issue is now resolved.

  • Update
    UTC
    Update

    A rollback to our primary node is required. This corrective action will take place this evening at approximately 18:30.

    A minor outage / increased error rate may be experienced for a brief period, whilst this action takes place.

  • Monitoring
    UTC
    Monitoring

    The underlying issue has been resolved.

    Our engineers are continuing to monitor the platform.

  • Update
    UTC
    Update

    We are continuing to work on a resolution for this incident.

    Updated list of affected services;

    • Automations and Custom Action processing
    • Jobs and Stock updates being shown in the UI
    • Outbound messages such as Job Details from Server to Devices

    Our engineers are on the case and another update will be provided shortly

  • Identified
    UTC
    Identified

    Our teams are currently actioning a failover between our Primary Nodes.

    A small outage might be experienced as connections to the database are switched between the SQL nodes.

  • Investigating
    UTC
    Investigating

    We are currently investigating an incident whereby the following actions are current delayed;

    • Automations and Custom Action processing
    • Jobs and Stock updates being shown in the UI

    Our engineers are on the case and another update will be provided shortly