Ghost in Your Supply Chain: Unpacking Cybersecurity Vulnerabilities in Software Supply Chain Management (Part I)

This post is based on the CyCraft research team’s presentation at the 2024 SANS DFIR Summit, titled “Ghost in Your Supply Chain,” which will be shared in two articles.

Through investigating two real-world cases, we analyze APT attacks targeting Taiwan via supply chains in 2023.

In the first case, attackers exploit vulnerabilities to invade the MDM platform, while simultaneously leveraging the project management system Jira to execute reconnaissance commands and collect information.

As for the second case, attackers conduct an island hopping attack against internal infrastructure of the financial sector, resulting in three subsidiaries compromised.

Lastly, we summarize our research findings on supply chain attacks and provide recommendations to strengthen their cybersecurity defense.

Obscure Defense Boundaries

Today’s IT architectures are increasingly complicated, making traditional boundary-based defense mechanisms such as intranets and firewalls insufficient. More dynamic and end-to-end application scenarios, including cloud services, BYOD, and WFH, have challenged the existing boundaries. Invisible security vulnerabilities within the supply chain further blur these defense boundaries.

Just as a modern city depends on complex supply chains for transportation, water, electricity, and food, IT systems also contain numerous supply chains. These supply chains embody trust relationships between enterprises, serving as shortcuts into corporate networks. For example, vendors may access a company’s internal network through VPN services to manage IT assets, or vendor software with cloud-based management and upgrade features may control endpoints inside the network. What’s more, the entrusted relationships between different enterprises within the supply chain makes it even harder to define and defend cybersecurity.

From the attacker’s perspective, the cost of direct intrusions significantly rises when MDR, SOAR, and threat intelligence are adopted. Therefore, identifying a vulnerable supplier within the vast supply chain and exploiting the trust relationship to penetrate target enterprises has become a more viable attack method. Moreover, in certain industries, most of the essential software or hardware are provided by particular critical suppliers. If attackers compromise one of these suppliers, they can simultaneously launch attacks on all of its customers, resulting in large-scale and high-impact security incidents.

In 2023, several major supply chain security incidents occurred:

1. 3CX Supply Chain Attack
In April 2023, the IT system of VoIP IPBX software developer 3CX was infiltrated. Attackers replaced the legitimate software of the company’s website with malicious versions containing backdoors. Mandiant’s investigation revealed that attackers gained their initial access through another 3CX supplier, Trading Technologies. This incident impacted the 3CX Desktop App, with nearly 12 million daily users.‍

2. MOVEit Transfer Managed File Transfer (MFT) Remote Code Execution Vulnerabilities
MOVEit, a software solution for encrypting and securely transferring files, is widely used around the world. In May and June 2023, cybersecurity researchers discovered numerous high-risk SQL injection vulnerabilities in MOVEit, which were actively exploited in attacks. The impact spanned across 32 countries and over 2,000 organizations. According to IoCs from different cybersecurity researchers, these vulnerabilities may have been abused even earlier. The Russian ransomware group Cl0p was found to have weaponized this gap and used it in real-world attacks.

3. Okta Customer Support System Credential Breach
In October 2023, identity and access management provider Okta disclosed a breach through which attackers compromised hundreds of customers, placing all user credentials at risk of exposure.

These incidents demonstrate how critical the supply chain is in modern cybersecurity. But what exactly is a “supply chain issue”? To answer this, the CyCraft research team conducts this deeper analysis into the root causes of supply chain incidents, classifies different types of supply chain threats, and develops corresponding strategies.

Classification of Supply Chain Threats

CyCraft's research team presented “Demystifying China’s Supply Chain Attack Targeting Financial Sector in Taiwan” at 2023 Japan Security Analyst Conference (JSAC). In this talk, we categorize supply chain attacks into four types (Figure 1).

Our classification is based on two dimensions: the initially compromised entity and the lifecycle stage of the compromised software. Concerning the initially compromised entity, attackers do not always target the victim enterprise directly. In many cases, they first compromise a supplier and then use it as a stepping stone to reach the final target. Hence, we further divide suppliers into system developers and service providers. As for the other dimension, we specify the lifecycle stage of the compromised software because vulnerabilities can be introduced and exploited at different phases, from development, distribution, to execution.

Strictly speaking, vulnerability in supplier’s software is not unique to supply chain attacks. However, since it remains unavoidable of the supply chain security discussion, we still include it in Figure 1.

According to Figure 1, the above cases fall into different categories:

• The 3CX supply chain attack belongs to the “Malware Implanted in Software”.
• The MOVEit Transfer Managed File Transfer (MFT) falls under “Vulnerability in Supplier’s Software.”
• The Okta Passwords Compromised is categorized as “Data Leaks from Out-sourcer.”

This shows that supply chain security is a diverse issue that cannot be simply addressed with certain solutions.

Software Supply Chain Threats

Among the four types of supply chain threats, “Malware Implanted in Software” and “Vulnerability in Supplier’s Software” specifically target the software supply chain. To analyze potential risks, we focus on the different stages of the software development lifecycle (Figure 2).

The complete software development lifecycle can generally be divided into three stages:

1. Develop: From source code to building the software.
2. Deploy: Distributing and installing the software on end-user systems.
3. Operate (Client-side): Executing the software on end-user systems.

Each stage consists of multiple steps, many of which may be susceptible to supply chain attacks.

During the development process, source code is the most fundamental raw material as well as the most fragile part. Many developers rely on official documentation, online resources, or reference books when writing codes. However, these references may prioritize functionality over security. As early as 2015, research has indicated that source code shared on Stack Overflow could contain security risks. Blindly copying and pasting such snippets may inadvertently embed vulnerabilities into the software (Figure 3).

One of the well-known code-related security incidents is the SolarWinds SUNPLOT supply chain attack (Figure 4). After breaching the enterprise, attackers inserted a backdoor directly into the software’s source code, embedding the malicious payload in its final version.

Libraries are another common attack vector in software supply chains. Since 2019, there have been cases of open-source libraries being injected with malicious code, such as the Event-stream incident. In 2023, the PyPI package management system even suspended new registrations due to the overwhelming number of malicious libraries (Figure 5), demonstrating the impact of compromised libraries.

The 2015 XcodeGhost incident is a classic example of a compiler attack. Attackers distributed the fraudulent Xcode tool which automatically injected backdoors in Apps. This shows that even compilers and other development tools can be exploited for supply chain attacks (Figure 6).

Supply chain attacks have also been observed during the deployment stage. For instance, in 2018, the system-cleaning software CCleaner was implanted backdoor, infecting over 2.27 million devices worldwide, including those in major technology and telecommunications enterprises (Figure 7).

Other supply chain incidents include ASUS ShadowHammer (2019) and the previously mentioned 3CX attack. Another example is the Jira Script Runner abuse incident which would be analyzed further in the case study section. Since systems like Jira are often integrated with CI/CD pipelines, these tools can become a channel for attackers to disrupt the CI/CD process. Moreover, we have identified a campaign dubbed Typing Ghost targeting software update channels. This case will be covered in detail in the following post.

Next, we will share two recently observed supply chain incidents to take a closer look at the nature of supply chain attacks.

Flowchart of Supply Chain Case Analysis

Case Study: Jira Script Runner Abuse

The CyCraft research team discovered this incident during the investigation of one of our clients, since CyCraft EDR detected suspicious webshell behavior in the environment. Jira is widely used for issue tracking, bug tracking, and agile project management. In this case, attackers used the tomcat web service to execute powershell commands, attempting to download malicious files. Under normal circumstances, web applications should not execute such commands unless explicitly designed to, making this behavior highly suspicious.

In addition, the client’s SOC detected that one VPN host continuously attempted large-scale scans against multiple internal web services. However, the web server involved in the incident did not provide external services. As a result, the client wanted to investigate the attack source, how the attacker leveraged the web service to launch powershell, and whether any vulnerabilities existed in the web service.

After investigation, we acknowledged that in this incident, the attacker first carried out large-scale scans against the VPN server of suppliers, in order to identify exploitable vulnerabilities or accounts. After gaining access to the VPN server, attackers then conducted extensive scans across multiple internal web services, during which they discovered a Mac MDM host that provided external services and was protected with a weak password (our client also admitted that this Mac MDM host had many unpatched vulnerabilities). After compromising this MDM host, attackers logged into the Jira web management interface using a weak administrator password. From there, they abused Jira Script Runner to execute powershell commands and download suspicious files. Although lateral movement was not successful in this case, abusing Jira Script Runner could have enabled further expansion.

Regarding the vendor’s VPN server that was detected performing large-scale scans on the internal web, we analyzed the VPN server’s Web Access Log and found that during the attack period, there was indeed an IP belonging to Alibaba (8.213.196.159) that had a high volume of access records targeting the VPN server’s webpage (Figure 10). After confirming with the client’s blue team, this activity was determined not to be part of normal business operations.

When analyzing Jira logs, we discovered that the attack originated from the aforementioned Mac MDM host. Therefore, we also examined the Mac MDM Web Access Log and found that the Mac MDM Web has also been subjected to extensive scanning, with the attack source again being the VPN server (Figure 11).

Since the Mac MDM manages Mac devices, it allows external users to remotely control hosts. In the Mac MDM Web Access Log, we observed entries labeled mdm_connect, which confirmed that someone has indeed accessed and remotely managed hosts through the MDM function (Figure 12). This reveals a significant management risk: if external access users are attackers, they could arbitrarily control multiple Mac devices.

Besides, when analyzing the Jira Access Log, we found numerous 302 responses. The “go http client” was identified as the agent used for the mass web requests, indicating that the scanning tool was written in Go (Figure 13).

In the call graph of this incident (Figure 14), we can see multiple commands launched via tomcat, such as starting powershell and CMD. This behavior is very similar to a webshell attack, where malicious commands are executed through web services. Common examples include web services such as w3wp for iis, httpd for apache, or PHP scripts used to run commands on a host. During analysis, we observed that the parent process web was used to launch further attack commands.

As shown in Figure 15, attackers executed malicious commands through web services, using tomcat to run ping for testing outbound connectivity, running reconnaissance commands such as whoami and dir, and executing powershell to download suspicious files.

Before analyzing the web logs of the Mac MDM and VPN servers, we had already observed the attackers were using powershell to test connections against the VPN server and using certutil from the Mac MDM host to download AD management tools (Figure 16). Thus, we were able to preliminarily determine that both the Mac MDM and VPN servers had likely been compromised.

When analyzing Jira’s security logs, we also found records of access through the VPN and Mac MDM servers. Moreover, the Jira administrator account had already been successfully authenticated, meaning attackers had logged into the web interface with this account. Aside from the admin account, attackers also accessed other Jira-related account information, presumably to discover additional accounts they could exploit (Figure 17).

Since we had earlier observed malicious commands executed through the tomcat web service, along with large-scale script runner execution events in Jira’s access logs, we suspected that this attack involved the abuse of Jira’s Script Runner. This was later confirmed when we discovered records in the web application log showing attackers running powershell commands to download suspicious files. We also found evidence of attackers executing “nslookup xxx.drxj5u6m[.]dnslog.pw” against a C2 server to verify DNS resolution. This is a common step in DNS tunneling attacks used to send and receive attacker commands. Through the Jira application logs, we confirmed that attackers had indeed used the Jira Script Runner to launch the attack.

Investigating Jira’s audit logs during the attack period, we found multiple instances of high-risk scripts executed by the attackers. Figure 19 shows an audit record where the admin account executed powershell to download a suspicious file, consistent with our earlier findings.

Thanks to early detection and rapid incident response, the attackers did not have the opportunity to use Jira for lateral movement within the internal network. However, from the technical perspective, such abuse would have been possible. We provide an example of this type of attack, along with how scripts can be crafted for the Script Runner in Figure 20 (replacing the red text in the right-hand box with commands listed on the left yields a usable script).

From this, it is clear that attackers could first conduct reconnaissance within the internal network, then execute credential dump to obtain valid credentials, and finally leverage tools such as psexec for lateral movement. To achieve persistence in the target environment, attackers might also implant Cobaltstrike malware, or deploy tunneling tools like shinysock.

The MITRE ATT&CK techniques and IoC list for this incident are as follows:

MITRE ATT&CK

Indicators of Compromise

- 139.180.159[.]74

- 45.76.188[.]249

- 8.213.196[.]159

- xxx.drxj5u6m[.]dnslog.pw

Conclusion

1. As IT architectures become more diversified and supply chain structures more complex, defense boundaries are increasingly blurred. Therefore, cybersecurity mechanisms must become more refined correspondingly. The CyCraft research team categorizes supply chain attacks into four types: Malware Implanted in Software, Island Hopping Attack, Vulnerability in Supplier’s Software, and Data Leaks from Outsourcers. Each of these attack types has different root causes and requires the application of distinct cybersecurity mechanisms.
2. Focusing on software development lifecycle and their components, we focus on case studies related to Malware Implanted in Software and Vulnerability in Supplier’s Software.
3. In the case of Jira’s Script Runner Abuse, attackers use the supplier’s VPN as the initial intrusion vector, which also belongs to the island hopping attack. This attack method is often overlooked in supply chain security discussions but is actually quite common in the real-world cases we have observed.
4. Project management systems like Jira are not only used as ticket tracking systems, but also possess certain execution privileges, enabling them to run commands on endpoints. This feature makes them vulnerable to persistence and lateral movement in post-exploitation stages. To defend against such threats, we recommend enabling Multi-Factor Authentication (MFA) on Jira.

For more details, please refer to CyCraft’s research blog: https://go.cycraft.ai/research-blog