Login Page - Request blocked when logging in

Incident Report for Total Synergy

Postmortem

Issue Summary:
Between 8:37 AM and 9:35 AM AEST on April 14, 2025, a platform change caused our system to apply login rate limits based on a shared gateway IP address, rather than individual user IPs. As a result, many valid login attempts were blocked once the shared threshold was reached.

Approximately 70% of users logging in via email/username and password were affected during this time. It did not impact users logging in via Single Sign-On(SSO) and social logins.

Resolution:

At 9:34 AM AEST on April 14, we temporarily increased the login rate limit to immediately restore access.
A permanent fix was deployed at 5:00 AM AEST on April 15, which now correctly identifies and applies rate limits based on individual user IPs.

The issue has been fully resolved. We are also implementing additional safeguards to help prevent similar issues in the future.

Posted Apr 22, 2025 - 11:56 AEST

Resolved

Between approximately 8:00 AM and 9:40 AM AEST on 14 April 2025, we experienced a login outage that prevented some users from accessing Total Synergy.

Our development team has implemented a fix, and users are now able to log in successfully. A permanent hotfix will be released in the coming days to prevent this issue from reoccurring.

We sincerely apologise for the disruption and appreciate your patience as we resolved the issue.

Posted Apr 14, 2025 - 11:37 AEST

Identified

Our development team have identified the problem causing the current login issues Total Synergy.

There will be downtime of up to 10 minutes from 9:35 AM AEST, we will continue to provide updates as they become available.

Posted Apr 14, 2025 - 09:34 AEST

Investigating

We're currently investigating multiple reports of users being unable to login this morning, 14 April, 2025.

Our development team are investigating this with high priority and we will provide updates as they become available.

This is only impacting Username & Password log-ins. Social log-ins (MS365 etc) are not impacted so please log in using alternative options where possible.

We apologies any disruption this is causing and we endeavour to have this resolved as quickly as possible.

Posted Apr 14, 2025 - 09:01 AEST

This incident affected: Synergy App.