Did NetEase Cloud Music collapse? Did they lay off the key people?

On the afternoon of August 19, 2024, NetEase Cloud Music suffered a major service disruption, causing its services on various platforms to be unusable. This incident quickly trended on hot searches and attracted widespread attention.

After the service disruption, many users reported trying various methods, including restarting their devices and switching networks, but the issue remained unresolved. Some even mistakenly believed that the problem was with their own devices.

Regarding this incident, there were rumors that it was caused by "internal staff deleting the database and running away," while others speculated that it was related to NetEase Cloud Music's recent server room relocation. However, NetEase Cloud Music officially denied both claims.

A senior R&D professional analyzed that the "downtime" of NetEase Cloud Music was more likely due to cloud storage operation and maintenance personnel being unfamiliar with the underlying system configuration and making mistakes during operations. This individual also pointed out that NetEase Cloud Music's response time was quite slow, with the repair taking two hours, while the usual repair time should be within 30 minutes. "This incident was definitely caused by unprofessional staff, and the backup system did not function as expected," they said.

Although it achieved profitability last year, NetEase Cloud Music is currently undergoing a structural adjustment period. In 2023, NetEase Cloud Music's revenue declined year-on-year, and the growth rate of paying users for its online music business, which accounts for a significant portion of its revenue, slowed significantly. This incident undoubtedly dealt a blow to NetEase Cloud Music and may have impacted its brand image and user trust.

After the incident, NetEase Cloud Music issued an official statement, stating that the infrastructure failure had caused services on various platforms to be unusable and that they were working to repair it urgently. Additionally, there were rumors online that it was caused by "internal staff deleting the database and running away," but NetEase Cloud Music quickly denied this claim, stating, "There was no database deletion, no one ran away, and the fault has been gradually repaired."

Some believed that the incident might also be related to NetEase Cloud Music's recent relocation of its server room to Guizhou. However, a senior R&D professional believed that the fault was not significantly related to the server room relocation and was more likely due to cloud storage operation and maintenance personnel being unfamiliar with the underlying system configuration and making mistakes during operations.

After about two hours of the fault occurring, some services of NetEase Cloud Music began to recover gradually. The aforementioned R&D professional pointed out that NetEase Cloud Music's response time was quite slow, and the usual repair time should be within 30 minutes.

"We generally use SLA (Service Level Agreement) to describe the available service time of a system in a year. The annual availability rate is usually 99.99%, and the annual downtime due to faults does not exceed three hours. When large websites encounter problems, the system itself will have a backup plan, which can automatically degrade and shield faults and quickly restore services. The downtime is only a few minutes at most. So the incident at NetEase Cloud Music was definitely caused by unprofessional staff, and the backup system did not function as expected," they explained.

It is worth noting that while NetEase Cloud Music was experiencing a crash, some users found that the NetEase News web page could not load normally and would redirect to the mobile homepage when accessing the NetEase News website. In response, the aforementioned R&D professional said that it was possible that NetEase Cloud Music and NetEase News were sharing the same cloud storage to improve resource utilization and reduce costs.

"Maybe they cut too deep during layoffs," the aforementioned R&D professional said. Some media reports claimed that in recent years, NetEase Cloud Music had undergone staff optimization to reduce costs and increase efficiency. Additionally, the Curve storage system team developed by NetEase had also experienced layoffs.

In fact, this was already the second time NetEase Cloud Music had experienced a "crash" this year. In mid-March this year, NetEase Cloud Music experienced a sudden failure of login status and could not be used normally. This incident also trended on hot searches, and NetEase Cloud Music's customer service later confirmed that it was caused by network anomalies and was not related to version updates.

For NetEase Cloud Music, service disruptions are not only technical challenges but may also impact its brand image and user trust. Especially after achieving profitability for the first time last year, this incident undoubtedly dealt a blow to NetEase Cloud Music.