So You’ve Decided to Hire an Application Security Engineer? Here’s what you need to know
Thursday, April 23, 2020
Posted by: Abbey Kwiat
Mike Zusman, CEO
Hiring your first appsec engineer is a high-risk endeavor.
Many organizations reach the conclusion they need to hire an appsec engineer after accumulating years – if not decades – of application security debt. Hiring an appsec engineer can be a long process, and for many organizations that don’t approach it correctly, is one that does not end successfully. This is important to point out because there’s an opportunity cost associated with waiting to address application security until you’ve brought on a dedicated resource. But even worse than not making any appsec hire is making the wrong appsec hire. A bad appsec hire can create a false sense of security for business owners, while also aggravating software engineers and alienating them from security as an engineering practice.
Appsec hiring is challenging because demand is high and talent is scarce.
We have a few clients who dream about hiring “appsec unicorns.” An appsec unicorn is someone who not only understands appsec fundamentals, but they also know how to write code, function as a member of a software engineering team, and can communicate effectively with all stakeholders in an engineering organization: software engineers, product managers, executives, and information security/risk managers. These unicorns not only fix tactical security bugs on their own, but they also act as leaders who increase the security engineering capabilities of the organization.
Appsec unicorns want to be part of a winning team, and they don’t like inefficiency.
I’ll be blunt: appsec unicorns don’t want to feel hamstrung by corporate bureaucracy, they get above market salaries, and they don’t want to waste time on a commute. If your organization does not have a progressive, modern approach to business and software engineering, you will have trouble recruiting and retaining not only a unicorn, but a capable appsec engineer. If you hire a good appsec engineer and expect them to commute 2 hours a day, every day, you can expect to have retention challenges due to the number of remote work opportunities available for great appsec people.
Do not rely on security certifications to help identify a good appsec engineer.
“Well, this person is a CISSP, and they’re the best candidate we have right now, so let’s hire them.” Widely accepted information security certificates are not a great indicator of success for an appsec role. Additionally, technical penetration testing certifications (such as OCSP or CREST) may be a better indicator of appsec fundamentals, but don’t indicate if a person can productively collaborate with software engineering teams. Putting a great penetration tester into an appsec role in an engineering organization more often creates conflict, rather than progress. This often happens due to a mismatch in perceived criticality of security issues. A pen tester may find what he or she believes to be a “critical” security bug, while engineering doesn’t believe it’s a security bug at all! Too often, the pen tester is exaggerating the criticality rating, and while engineering may be forced to begrudgingly fix whatever the issue is, more damage has been done to the team than risk actually reduced for the organization. What looks like a tactical win for security actually introduces additional business risk because engineering will be less like to cooperatively engage with security in the future.
An efficient hiring process will help you hire the right person, faster.
There are high-end recruiters who focus on information and application security (contact us for recommendations), but most recruiters just don’t get it. “Wait, you’re hiring a security person, but you want them to be a software engineer too?” Recruiters often look for candidates that have security certifications as well as pen testing/security tooling experience. This is a trap, and you will be likely flooded with unqualified candidates for your appsec role.
You need a fast and easy screen to weed out unqualified candidates as early as possible.
At Carve, we use a series of fun reverse engineering puzzles to see if a candidate has fundamental computer science/engineering instincts required to be a good appsec engineer. The puzzles are of increasing complexity, and culminate with our software-driven Work Product Test (WPT). WPT is a complex and challenging technical exercise designed to simulate our most common appsec engagements. Those who enjoy and complete WPT successfully, in addition to having the technical chops to succeed as an engineer, are also motivated and excited at the possibility of joining our team. Only when candidates invest their own time into our WPT and are successful do we invest significant time into that candidates interview process. Building out puzzles and your own WPT might not work for you. The next best thing: let your software engineers interview your appsec candidates. After all, if your software engineers don’t like your appsec engineer, your new hire will do more harm than good.
Once you identify candidates who have the required technical chops, you need to assess their ability to work as an individual contributor, as a member of an engineering team, and as a leader in your organization. If you find someone who can do all three, you may have an appsec unicorn – you better move fast! Otherwise, be prepared to structure the role based on the candidates strengths.
One engineer might not be enough; you might need an appsec team.
Appsec unicorns, aka the single person appsec team, are hard to find, hire, and retain. To compensate, be prepared to design and build an application security team. For example:
- Application Security Director: reports to CTO/SVP; peers with VP of Eng/Product; works with the business and technology leaders to understand the business direction, and create a supporting appsec strategy
- Security Architect: reports to appsec director; works with software architects to identify security assumptions and prescribe architectural security controls; advocates for security in the engineering organization
- Security Engineer 1: focused on building/implementing re-usable security code and features, for example: secure APIs, authentication/authorization controls, encryption code & utilities
- Security Engineer 2: focused purely on vulnerability identification and remediation through code review, internal penetration testing, and third party penetration testing
- Security Engineer 3: focused purely on security infrastructure (not infrastructure security) to automate detection and remediation for classes of security vulnerabilities; includes internal tool development and evaluation/acquisition of commercial and open source security tooling
An application security program with strong leadership will have an impact that outlasts any single appsec hire.
Even at great companies, employees come and go. For long term application security assurance, you need an appsec program that is bigger than any single person. Long term success starts with making security a strategic priority for your software engineering organization. Bringing in a strong security leader with software engineering experience will shift the perception of security across engineering teams, and start them down the path of building purposefully secure applications and systems. A limited number of appsec engineers can be more effective when the bulk of your software engineering org is capable of handling low-hanging fruit security bugs themselves.