Software (in-)security in Germany

In my years working in application security teams for leading tech companies such as Facebook, I have identified several recurring patterns that challenge the effectiveness of cybersecurity measures in European software companies.

In this blog post I will provide some of the bad patterns that I have seen over the recent years, and where I believe you should rather invest your money. It may sound a bit raging, but it did provide some therapeutic effect on me writing it out 😉

1. Overreliance on certifications
2. Penetration testing
3. Security as a checkbox
4. Top-down management
   1. Strategic decisions
   2. Career progression
5. Required changes

Overreliance on certifications

Two years ago we performed a technical application security review for a healthcare provider in Germany, and within 30 minutes of the review we were able to login as arbitrary user – without a password.

The bug itself was simple and trivial, they basically hard-coded a signing key in their software and this key was used for signing authentication assertions. Anyone with posession of this key is able to generate valid authentication assertions as anyone, and the key was the same for any installation. (it’s kind of similar to a JWT assertion, but instead of JWT they have completely implemented their own scheme)

Despite the healthcare provider investing significantly in their software licensing, it was disappointing to find such a basic flaw. But it proved surprisingly difficult to get in touch with the rights stakeholder of the major German software vendor as they didn’t believe this bug to be real.

Eventually, we found ourselves on a call with their CEO. Regrettably, the discussion veered off track quite swiftly when the CEO began asserting outright that their security was superior to that of Facebook:

You may be good enough to find security bugs at Facebook, but that doesn’t mean you are good enough to find bugs in our software.

CEO of a german healthcare software company

After patiently reiterating that we provided a proof of concept video, script, and Windows binary, the CEO responded with unexpected skepticism:

We pay over 200,000 Euros a year for security certifications. The bug that you have found is fake! We are secure.

CEO of a german healthcare software company

After enduring further skepticism, we were eventually connected with the appropriate individuals in their engineering team, who acknowledged the bug and proceeded to fix. They also called all their customers and urged them to apply the security patch.

But this right here shows the issue: Spending hundreds of thousands of Euros in Snake-Oil. The CEO really believed that because he spends so much money every year on certifications from some famous German certification authorities, would insure their software is secure. But all these tests do is running automated scans and following checkboxes!

Having found this bug within minutes shows how relatively insecure this software was, and continues to be – but that is the unfortunate choice of this software vendor).

Do’s:

• Integrate secure by design frameworks: Implement frameworks that make it nearly impossible for your developers to create security issues.
• Invest in a dedicated application security team: Ensure your team consists of skilled engineers with a deep understanding of your technology stack and web/native/mobile security issues.
• Consider professional help: If you need assistance in building such a team or improving security in your company, seek out a professional consultant.

Don’ts:

• Avoid over-investment in checkbox security certifications: While certifications can be beneficial, relying too heavily on them can create a false sense of security.
• Don’t equate security certifications with guaranteed software security: Recognize that a certification is just a starting point and does not necessarily indicate that your software is entirely secure. Regular updates, monitoring, and improvements are crucial.

Penetration testing

We really care about security and perform quarterly/yearly security audits.

Having worked for security consulting companies, this is a sentence that I have heard many times and they always put my mind at unease.

A penetration test does not prove your software is secure. It rather proves that the security company wasn’t able to find more security bugs in the given time in the current software version. But very often the time allocated to pentests is just way to short.

I have witnessed instances of 10-day penetration tests for complex software, where setup alone consumes up to 30% of the allocated time, with reporting consuming another 10-20%. This leaves precious little time for the actual security evaluation.

This ultimately means the pentester will be able to spend 5 days on finding bugs in your software. This is rarely enough to really understand the application domain completely and identify most of the critical vulnerabilities.

And to be honest, I have seen many security pentest companies which just perform automated scans and charge horrendous amounts for that. So be very worrisome of some of these actors.

Do’s:

• Set up automated security scanning: Implement your own automated security scans instead of relying solely on expensive security companies. This will allow you to regularly monitor your systems for potential vulnerabilities.
• Implement holistic security measures: Whenever a security issue has been discovered, go beyond just fixing it. Find a comprehensive approach to prevent similar issues in the future from recurring.
• Use external security auditing wisely: Engage with external security companies on a long-term basis to maintain domain knowledge and consistently review new features.
• Prioritize white box security reviews: These reviews grant the tester access to your source code, providing a more effective and efficient way to identify security issues compared to black box reviews.

Don’ts:

• Avoid one-off engagement with pentest companies: Engaging pentest companies on a one-time basis may not guarantee the continuous security of your software.
• Beware of blackbox reviews or lack of domain-specific knowledge: Ensure your pentest companies have the specific expertise for your tech stack and provide thorough white box reviews.
• Don’t limit your security efforts to automated tools: While automated tools are a crucial part of a secure development lifecycle, they cannot replace the need for a skilled security team and continuous learning.

Security as a checkbox

Security is too frequently reduced to a simplistic checklist in an Excel spreadsheet, a perspective that can undermine its complexity and importance. This is especially a symptoms of all these certifications that don’t necessarily make the software more secure, but give everyone this warm and fuzzy feeling of security. (“Passwords must not be stored in plaintext: Check”)

Don’t get me wrong, these checklists have a purpose as an absolute baseline. And if you don’t deal with custom software, then they are usually fine. But as soon as it comes to application security these checklists don’t necessarily provide any meaningful value.

Consider the case of a custom-made financial application, which may be unique in its transaction handling, user data management, or encryption methods. A standard checklist would not account for these unique attributes and could overlook vulnerabilities specific to the application’s custom features. In this scenario, a more comprehensive and customized approach to security would provide far more meaningful value than a general checklist could offer.

Do’s:

• Understand the complexity of security: Recognize that security is not a mere checklist but a complex field requiring proactive measures and a deep understanding of the application’s vulnerabilities.
• Implement a Secure Development Lifecycle (SDLC): This involves integrating security practices into every stage of the software development process.
• Go beyond checklist security testing: Opt for comprehensive security testing, which includes vulnerability scanning, penetration testing, and code reviews, instead of merely ticking off items on a checklist.
• Invest in continuous education: Provide training to developers and employees on secure coding practices and general security awareness to improve the security culture within the organization.

Don’ts:

• Don’t solely rely on checklists: Although they serve as a starting point, checklists don’t guarantee comprehensive security. Supplement them with thorough testing and continuous monitoring.
• Avoid over-reliance on certifications: Certifications can provide a baseline level of security, but they shouldn’t be your only indicator of software security.
• Don’t neglect ongoing security monitoring and testing: Regular assessments are essential for identifying potential threats and vulnerabilities.
• Never dismiss security vulnerabilities: Address and fix all vulnerabilities promptly, regardless of their perceived severity. Ignoring seemingly minor issues can lead to major security breaches.

Top-down management

Strategic decisions

In many European companies, there exists a prevailing culture of top-down management. This approach not only impedes the growth and acknowledgment of technical expertise, but it also detrimentally impacts software security. Decisions about security protocols and policies are often dictated by managers who may not have the same depth of understanding as those with technical expertise. This type of management style overlooks the invaluable insights that technical experts, intimately acquainted with the evolving landscape of security threats, can provide.

Moreover, when project roadmaps and timelines are strictly controlled by managers, this can stifle innovation and responsiveness. The potential for technical teams to identify, explore, and address emerging security concerns becomes severely limited. This rigidity can result in slow implementation of critical security updates, consequently diminishing the overall security posture of the organization.

Career progression

This management issue also impacts career progression. In companies like FAANG, where technical expertise is highly valued, individual contributors can advance their careers without having to transition into management positions. This parallel career track encourages technical excellence and allows individuals to deepen their knowledge, ultimately benefiting the organization’s security posture. Conversely, the top-down management approach in many European firms stifles the growth of technical experts. Talented technical professionals are often forced to choose between staying in their current roles and sacrificing career progression, or transitioning into non-technical positions solely for the purpose of climbing the corporate ladder.

To address these challenges, companies should consider adopting a more collaborative and flexible approach to management. Security insights from technical experts must be valued and integrated into decision-making processes, significantly enhancing the organization’s security outlook. It’s equally critical to incorporate flexibility in project roadmaps, enabling proactive responses to emerging security threats. In an era where cybersecurity threats are continually evolving, management practices must adapt to stay ahead.

Do’s:

• Value technical expertise: Utilize the invaluable insights from your technical experts, as they are often more intimately acquainted with the security landscape and its evolving challenges.
• Adopt a collaborative approach: Integrate a more cooperative management style that values input from all team members, particularly those with hands-on technical experience.
• Allow flexibility in roadmaps: While planning is essential, allow room in project timelines for addressing emerging security issues and updating security measures proactively.
• Establish a parallel career track: Provide opportunities for technical professionals to advance their careers without having to move into management positions. Encourage and reward technical expertise and excellence.
• Integrate security insights into decision-making: Incorporate technical team’s security recommendations into your company’s strategic decisions to enhance overall security outlook.

Don’ts:

• Dictate security protocols from the top: Managers without technical expertise should not be the sole decision-makers for security protocols and policies. These decisions should involve inputs from the technical teams.
• Stifle innovation: Don’t let rigid management styles inhibit the ability of your technical teams to explore innovative security measures.
• Ignore emerging security concerns: Ensure that you’re not solely sticking to the plan at the expense of addressing new and pressing security threats.
• Force technical professionals into management: Don’t make your skilled technical staff choose between advancing their careers and maintaining their technical focus.
• Neglect the value of technical insights: Do not dismiss the inputs from your technical team. Their hands-on experience with the software and understanding of its vulnerabilities are critical to improving your security stance.

Required changes

To put it all together, enhancing software security in German, and indeed all European companies, requires a paradigm shift in management style, a re-evaluation of the value of certifications, a more nuanced approach to penetration testing, and a move beyond seeing security as a mere checkbox.

These are complex challenges, but by adopting secure by design frameworks, prioritizing continuous learning, and fostering a collaborative culture that values technical expertise, we can navigate them successfully. After all, the stakes in cybersecurity have never been higher, and our response should match the urgency and complexity of the threat landscape we face.