Web Application security is hard, especially when you are considering all the paradigm changes involved in running cloud applications. It turns out that even the so-called “experts” can decide to overlook things – Is your organization able to respond if a problem like this is found?
When DevOps pipelines leak in public
The Lionfish group have been around for quite a while, and as a result we have connections to nearly everybody in our industry, whether it be top CEOs, senior CISOs and CIOs, or even security researchers spelunking through the digital underground. A friendly white-hat who goes by the name of Gebo Wyrd told us of a fascinating discovery with a vendor called Noname security. Noname supply products designed to improve security of APIs, which have become a critical background in the struggle to keep data secure.
Gebo showed us a video which portrayed the Jenkins server that Noname use is widely accessible from the internet. As per their wiki “Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron.” A little research at sites such as PentestGeek or HackTricks Cloud shows that an exposed Jenkins server can lead to theft of secrets, perturbation of builds, and many other highly undesirable impacts in the DevOps process. Perhaps Jenkins should pay a bit more attention to client side security as well.
No delays, no problem
NoName have established a bug bounty program where they offer to recognise and reward security researchers who find and report problems in good faith. Gebo told us that he was worried as despite sending notifications through the program and via the 80 or so internal emails he discovered, he had not seen any response. Given the accessibility of tools such as Shodan, and that researchers (both Black and White Hat) are using it to find open Jenkins instances, Gebo was concerned that Noname could have a high-impact incident develop. (In fact, tools such as Shodan are not necessary. A simple Google Search reveals 91 open Jenkins servers (at time of writing), some of which are of a similarly high profile to Noname.)
Noname’s open Jenkins server allowed Gebo to find dozens of internal emails, email aliases for group tasking etc…. Even though Gebo sent this information to everyone on the emails that were discovered, the server remained online for weeks.
The Jenkins server was building custom environments that are customer keyed, the orchestration also aided in gathering data from Noname’s primary code repository and executing software builds amongst other support and customer service functions. Living off the land and using the API Keys that can be extracted from Jenkins means an attacker could have been infiltrating their source code, reading their customer service applications as well as instrumenting themselves into their customer environments with root credentials.
Having had the report from Gebo, and received confirmation that the server was exposed from other sources, Lionfish reached out to Karl Mattson, CISO of Noname and warned him of this vulnerability in their DevOps pipeline process. Despite being in travel, Karl responded with agility and precision, as with any resource open to the public it is simple to pull the subdomain down from public view. He assured us that the finding was in hand and would be dealt with rapidly, and very soon the errant Jenkins server disappeared from the internet. Karl asked us if we wanted a reward from the bounty program which was kind, but this wasn’t about the money for us – it was about making sure that the Internet, and the critical software and systems that enterprise rely on for their cybersecurity, are a little bit safer.
What can we all learn from this?
There’s one thing that is very clear. If an experienced, reputable organisation like Noname can be exposed through a slip in configuration, we all can. The massively connected digital world has many unexpected gaps where an attacker can slide in and wreak havoc – or perhaps worse, silently wait until the right moment to strike. We know that some banks that were impacted in the 2015-2016 cyberattack on SWIFT-connected banks still report that they believe themselves to be infiltrated by highly skilled, determined adversaries.
Secondly, there’s the question about private bug bounty programs. A lot of researchers in the field report difficulties when reporting vulnerabilities through “homebrew” programs. (Having said that, many report that the bug bounty companies aren’t always a picnic, either.) But clearly whether a bug bounty program is run internally, or outsourced to a third party agency, it must be seen as a trusted broker between the reporting researcher(s) and the organisation, in the same way that an ethics reporting line or whistleblowing program must be. The jury is still out on whether this is best achieved internally or externally. If a security company doesn’t do security reports well, I guess it will be hard for anyone. Nevertheless, when Gebo finally tracked down someone the CISO knew, had a problem brought to his attention, he acted swiftly and decisively.
Thirdly and finally, the role of a security researcher is fraught with difficulty. There’s the pressure to find the vulnerabilities, the time-consuming nature of the research (and consequences for family and social life), and then the worry that your findings will be dismissed, or worse still, silently ignored. Of course, there’s always the fear that a company whose pride has been wounded will choose aggression and litigation rather than cooperation and collaboration. It’s for this reason that many researchers will retreat behind a pseudonym, often becoming well known in the community by that pseudonym alone.
So, what can you do?
Ultimately, this can be viewed as a live-fire Zero Trust exercise. Do not trust that your operations teams have deployed resources with the correct permissions, whether they be data buckets, DevOps pipelines, or customer-facing apps. People make mistakes, err on the side of “trust”, belief in obscurity, and are always overworked and rushed.
When an error is pointed out, resist the urge to lob a retaliation strike. Check out the report, and assign it the priority it truly deserves. Like any incident, vulnerability reports are a rich learning opportunity for your team.
And most of all, anticipate incidents. Have a well-thought-out, properly rehearsed response plan so that when something does hit the fan, your team are not over-adrenalized but treat this as just another day in the amazing world of SecDevOps.
— Jonathan Care, Advisor, Lionfish Tech Advisors Inc.
Header image sources: Wallpaper Flare, Unsplash