RealConnect Reporting Delayed

Incident Report for Poly Cloud Services

Postmortem

On May 26th Poly identified an issue with the primary database index used to order call event information for the RealConnect service. While this issue was corrected later that evening, it resulted in a sizable backlog of data to be ingested into the database. Reducing this backlog while minimizing any call event data loss, took a significant effort. By June 10th, the backlog had been successfully reduced and the reporting is back to its normal operating behavior. Periods of missing historical call events are being addressed.

Posted Jun 14, 2022 - 15:54 MDT

Resolved

This incident has been resolved.
Posted Jun 13, 2022 - 14:26 MDT

Update

Poly will continue to monitor RealConnect over the weekend. Next update will be Monday AM.
Posted Jun 10, 2022 - 15:19 MDT

Monitoring

The lag in call events has been eliminated. Poly is working on ingesting the outstanding historical call information.
Posted Jun 09, 2022 - 21:22 MDT

Update

The lag in call events continues to be reduced, but a small lag still exists. Poly is working on ingesting the outstanding historical call information.
Posted Jun 09, 2022 - 19:44 MDT

Update

The reporting system remains stable. The lag in call events continues to be reduced, but still exists. Poly is working on ingesting the outstanding historical call information.
Posted Jun 09, 2022 - 06:43 MDT

Identified

Poly continues to work the issue. The reporting system remains stable. As of this update time the lag is within minutes of real time. Poly is working on ingesting the outstanding historical call information.
Posted Jun 08, 2022 - 16:46 MDT

Update

Investigation efforts continue towards reducing the reporting lag.
Posted Jun 07, 2022 - 17:08 MDT

Update

The lag in reporting has been reduced but still exists. Poly is actively working to ingest the outstanding call information into the database. There is no impact to calls.
Posted Jun 06, 2022 - 15:22 MDT

Update

The lag in reporting has been reduced but still exists. Poly is actively working to ingest the outstanding call information into the database. There is no impact to calls.
Posted Jun 06, 2022 - 06:58 MDT

Update

There continues to be a lag in reporting. Poly is actively working to ingest the outstanding call information into the database. There is no impact to calls.
Posted Jun 05, 2022 - 18:35 MDT

Update

There continues to be a lag in reporting. Poly is actively working to ingest the outstanding call information into the database. There is no impact to calls
Posted Jun 05, 2022 - 08:22 MDT

Update

There continues to be a lag in reporting. Poly is actively working to ingest the outstanding call information into the database. There is no impact to calls
Posted Jun 04, 2022 - 19:26 MDT

Update

Poly continues to troubleshoot the delay in ingesting the outstanding call information into the database. There continues to be a lag in reporting. There is no impact to calls.
Posted Jun 04, 2022 - 07:49 MDT

Update

Poly continues to investigate the delay in ingesting the outstanding call information into the database. There continues to be a lag in reporting. There is no impact to calls.
Posted Jun 03, 2022 - 16:49 MDT

Update

Poly continues to focus on ingesting the outstanding call information into the database. There continues to be a lag in reporting. There is no impact to calls.
Posted Jun 03, 2022 - 05:52 MDT

Update

Poly continues to focus around ingesting the outstanding call information into the database. The continues to be a lag in reporting. There is no impact to calls.
Posted Jun 02, 2022 - 17:15 MDT

Update

Poly continues to work the issue. The work continues to focus around ingesting the outstanding call information into the database.
Posted Jun 02, 2022 - 05:58 MDT

Update

Poly continues to work the issue. The work continues to focus around ingesting the outstanding call information into the database.
Posted Jun 01, 2022 - 17:17 MDT

Update

Poly continues to work the issue. the team continues to work on ingesting the outstanding call information.
Posted Jun 01, 2022 - 12:50 MDT

Update

Poly continues to work the issue. The reporting system has been stabilized. It is currently several hours behind and we are working on ingesting the outstanding call information.
Posted May 31, 2022 - 17:04 MDT

Update

The RealConnect devops team continues its stabilization efforts for the reporting system and is working to reduce the data lag.
Posted May 30, 2022 - 15:05 MDT

Investigating

We are currently investigating this issue.
Posted May 29, 2022 - 16:04 MDT
This incident affected: Poly RealConnect (NALA, EMEA, APAC, RealConnect Tenant Report).