listen to this article:
Code signing proves the identity of the source vendor of a software (only the private code signing key holder can create the signature) and verifies that the code has not been tampered with since being published. Code signing can be done in two phases, at the first once binary files (library/executables) are signed, and in the latter packages containing collections of such (e.g. RPM, docker, NuGet) are signed in order to provide means to validate the released packaged code.
Growth and Complexity of Code Signing
Requirements for code signing will grow as single monolithic apps are replaced by multiple microservices developed by disparate teams, service velocity leads to more frequent releases, and DevOps processes will include code signing in varied stages of the lifecycle prior to release. As code signing grows in both importance and complexity as a security best practice, there also exist challenges in implementing secure code signing solutions within organizations.
DevSecOps Approaches to Code Signing
There exist a number of code signing approaches, and each approach has its respective challenges.
Local Code Signing Approaches
Unmanaged & Unsecure
- Private keys are placed in signing servers or users’ endpoint devices – not in secure storage, creating a very real possibility of signing keys compromise.
- No centralized key management, no administrative control – everything must be done manually.
- No capability to enforce security policies consistently.
- HSMs are difficult to manage, and DevOps teams that are in charge typically do not possess the capacity to manage an HSM.
- Using HSMs would require a very large number (due to the number of the build servers in an organization) and variety of clients (due to the need to support various platforms), making HSM usage a real nightmare of constantly resolving signer integration issues. This can be avoided by taking the “Unmanaged & Unsecure” approach, which creates other problems (latency and security issues with sending the files to be signed to the central signing server); the comparison of these approaches will be discussed below.
- Key usage control is a security issue (the code signing keys are well-protected in the HSM, but such solutions don’t allow to control what is signed).
- No ability for supporting ephemeral clients (e.g. signing from containers).
- This is not a product working OOB and requires implementing integration with each of the required signers/development infrastructures.
- Large organizations have difficulty synchronizing keys across different geographies.
- Not supporting distributed architecture.
- Very basic support for file types (mostly MS), cannot sign RPM or Mac.
Remote Code Signing Approach
- Current leading solutions have no integration with public CAs, making the certificate purchase and renewal cumbersome, and having to use an additional tool for certificates management.
- No built-in key usage control measures – this is especially important as we have seen a number of hacks (as in the case of ASUS, see more here) where the hackers succeeded to use the existing code signing solution to sign code that included malware.
- Requires managing HSMs, as it’s not an integral part of the code signing solution.
- Lacks proper client security.
- No integration with signing tools/build servers.
- Limited platforms (mostly MS).
- If signing large files is required there is a need to keep on-prem instance (for performing the hash operation).
- Lack of filetype support can make the service irrelevant, dependent on the specific list of file types each service provider supports.
- Integration only with one public CA (this is the case as the only SaaS vendors in the market currently are CAs).
- The end customer cannot manifest control over the code signing keys as the keys are held by the service provider.
Local vs Remote Code Signing
When evaluating a code signing approach, an often overlooked but nonetheless important consideration is its ability to perform local and remote code signing. One of the specific issues that we would like to delve into is a comparison between the local and remote signing options. For example, the “Unmanaged & Unsecure” and “HSM” approach only allow for local code signing, whereas the SaaS approach only does remote (while also offering local due to performance issues). A dedicated end-to-end code signing solution can do both, depending on the implemented topology.
|Parameter||Local Signing||Central / Remote Signing|
|Performance||The hash is created locally, while the signature is created at the secure location of the code signing keys storage (i.e. in vHSM/HSM), distributing the overall required performance across all the building servers in the organization, across different times, no delay penalty.||Transferring large files from the local build servers to the location of the centralized code signing system can cause a substantial delay, hindering the release cycle. It also requires a centralized solution that can cope with the required performance, compared with the distributive nature of the local signing.|
|Integration with signers* and development environments||If the signing is done locally there is a seamless integration between the signer and the security provider. IDE** such as Visual Studio that utilizes local signing are extremely difficult to allow centralized code signing due to the need to perform an integration with the security stack of the machine that the client is installed on (i.e. JCA*** , MS KS**** )||If the signing is performed remotely, there is a need to have all the relevant platforms (i.e. all types of Linux distributions, Windows servers versions, Mac, etc. that exist in the organization in a central location), and manually configure the security providers for each signer.|
The End-Customer Perspective: Integrating Code Signing in DevSecOps Processes
R&D and DevOps leaders are looking to automate, speed, and simplify the code signing process. Code signing must be integrated within the DevSecOps processes and enhance it by improving operational efficiency. Different development teams might have different SDLC flows, in which the code signing solution must naturally fit, without causing a disturbance in the development process. A proper code signing solution should support the existing platforms and development environments, as well as respond quickly to changing market demands – support for additional signers, platforms, and development environments (microservices, containers).
It is essential for all the relevant stakeholders in the enterprise that are involved in code signing operations (such as CIO, VP R&D, CISO, Head of DevOps) to have a holistic overview of the code signing process – what is signed, on which platforms, what type of certificates are used (OV/EV), how the certificates are managed, how the code signing keys are protected, how the entire solution is managed and automated, who have permissions to do what, how the users are authenticated, etc., and come up with the most suitable course for the organization.