Massive Salesforce Outage Resolved With Gradual Access Restoration

Clients affected by the Pardot marketing automation software scripting error received System Administrator access Saturday, but still needed to figure out how to return profiles and user permissions to their previous settings.

Salesforce users slowly regained access to the company's tools over the weekend following a massive service outage Friday.

The San Francisco-based cloud software giant Friday blocked access to all current and former Pardot marketing automation software clients after the deployment of a database script inadvertently gave users broader data access than intended.

Although the scripting error only affected the Pardot service, Salesforce initially blocked access to all instances of its software for affected organizations, according to an article on the company's website. Once Salesforce was able to isolate the affected organizations, the company said it restored access for non-affected customer organizations.

[Related: 'Major' Salesforce Outage Whacks Firm's Marketing Automation Customers]

Salesforce said that, as of Saturday, customers who were unaffected by the database script issue had their full access restored. For clients affected by the database script issue, though, Salesforce on Saturday only restored access to users with a System Administrator profile.

The company said it is continuing to work to restore permissions for affected organizations to where they were prior to the database scripting error. Now that Salesforce has restored administrator access to all affected organizations, the company said businesses can restore their profiles or user permissions.

Organizations with a valid backup of their profiles and user permission data can deploy that information directly from a Sandbox copy to the production environment, Salesforce said Saturday.

But if a customer is looking to restore its previous settings and lacks a Sandbox containing production profiles and permission sets, Salesforce said Saturday that admins would have to manually modify the configurations to grant appropriate access to users.

Automated provisioning to restore permissions has been executed on all production instances, Salesforce said in an early Monday update. A subset of customers may still be experiencing issues with user permissions, and Salesforce said its teams are continuing to work on this.

Salesforce users across North America and Europe found themselves flummoxed Friday by the lack of access, with the number of reported outages peaking at 3,262 shortly after 1 p.m. ET that day.

"To all our @salesforce customers, please be aware that we are experiencing a major issue with our service and apologize for the impact it is having on you," Salesforce Co-Founder and CTO Parker Harris wrote on Twitter at 12:40 p.m. ET Friday. "Please know that we have all hands on this issue and are resolving as quickly as possible."

Users started airing their grievances Friday afternoon under the hashtags #salesforcedown and #permissiongeddon, both of which were still being used well into the evening Sunday.