Introduction

While working on a server for a client recently, I had the opportunity to take on an exciting challenge, albeit different from my usual projects. While I will preserve the client’s anonymity in this article, I want to share this experience which highlights the importance of technical expertise and a methodical approach to solving complex problems.

Following a distress call from a panicked client, we at SQLI were asked by the owner of a car garage franchise to investigate a major malfunction in their server application. Since we had never worked with this client before, we decided to go on site to assess the situation and determine the feasibility and cost-effectiveness of the intervention, as mobilizing a team of ten people over an extended period of time required significant resources, potentially out of reach for the client.

Analysis of the problem

Once we arrived on site, we discovered the story behind this critical situation. Following a dispute between the boss and the employee in charge of developing and managing their in-house CRM, the latter had left the company, leaving the CRM suddenly dysfunctional. This scenario seemed too unusual to be a mere coincidence.

The objective of the intervention was clear: connect to the server and fix the CRM. However, this task was going to be more complex than expected, as all the information related to the administration of the server was held by the departed employee, and the boss had no means of access, no SSH credentials or passwords. Although it seemed daunting, we agreed to spend an entire day looking for a solution. We were aware that our efforts might not be successful, but the hope of finding a solution persisted.

Fortunately, the server was hosted by Infomaniak, which allowed the client to create an account for me within his team. This gave me access to the server’s IP address, the first crucial step in my quest for access. The first attempt was to contact Infomaniak’s technical support in the presence of the customer in order to reset the default identifiers, but this approach was unsuccessful and we were asked to resolve the problem ourselves.

Faced with this impasse, I considered a more crude approach. I wondered if rainbow tables containing SSH passwords could be used to reveal the correct password. Knowing that the default account was “debian”, I already had the username, but I was still missing the password. Unfortunately, after a few unsuccessful attempts at common passwords, it became apparent that the server only accepted connections via an SSH key. A new difficulty to overcome.

Perseverance and solution

Despite the seemingly unsolvable nature of the problem, I informed the client that I was determined to continue my research until the end of the day. Then, suddenly, I discovered a solution right at my fingertips that had been in front of me all along. The Infomaniak administration interface allowed me to switch the server to recovery mode. In this mode, another almost empty server is activated, with identifiers provided by Infomaniak, and the hard disk of the original server is available separately to be mounted and loaded into this temporary server.

So I switched to recovery mode, used the identifiers provided by Infomaniak, and bingo! I was able to access a temporary server! All I had to do was mount the target server’s hard disk and explore the files. I quickly identified the profile of the employee concerned and replaced his SSH key with my own. Then I restarted the server in it’s normal state… and it worked! I was able to connect to the server using the employee’s account and my SSH key!

Results

The first action I took was to review the history of executed commands, to gather information for my report. The last commands executed were pushing all the CRM code to the employee’s GitHub account, before deleting everything from the server. This behavior explained why the CRM was no longer responding.

I investigated the server further, gathered all relevant information, including discovering a separate website, which was an old, highly customized version of WordPress, with database identifiers present in clear text in about 50 custom files. This suggested to me that the quality of the CRM code must not be optimal. To mitigate the risk, I changed the database identifiers as well as in each file, preventing the employee from causing further damage. I then finalized my report, including all incriminating evidence, as well as new credentials for each database and server account, which I gave to the client. Although he was unable to retrieve his CRM code, he had all the information he needed to take further action, if necessary.

Conclusion

A few days later, the client spoke with the employee in question and was able to retrieve his CRM code, which was reinstalled on the server. Afterwards, he asked us to take over the development of the CRM for the employee. However, after conducting a thorough audit of the quality and maintainability of the existing code, we expressed doubts about the feasibility of this takeover. The CRM code was extremely complex, lacked consistent logic, and it was surprising that it still partially worked. We advised the client to rebuild the CRM from scratch, following standards and best practices, to ensure a stable, maintainable and future-proof solution. Since then, we have not heard from the client. I hope that he was able to complete his CRM in time and that he has taken steps to properly back up his data.

This intervention on a client’s server was a rewarding experience, highlighting the importance of deep technical expertise and a methodical approach to complex situations. It also highlights the importance of information security and taking preventive measures to prevent critical data loss.