Cutting Corners
I’m convinced cutting corners is inextricably linked to how humans function. It’s like you’re attracted to cutting corners, instinctively following a desired path. Yes, I plead guilty. But before cutting a corner, I always go over the different solutions considering the tradeoffs of each option. That’s my process to find the right balance between cost and benefit.
From time to time, you stumble upon a solution you thought you’d well consider. When, a few weeks later, giving your solution a second thought, you’re fazed. That’s what happened to me in this case.
Get to the point
One of the first things you learn on AWS is to follow the standard security advice of applying least privilege. It’s a simple rule, but it can be challenging to implement. When creating an AWS CodePipeline a while ago, we came up with the following CloudFormation template describing the infrastructure:
DeliveryPipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
ArtefactStores:
- Region: eu-west-1
ArtifactStore:
Location: !Ref ArtefactBucketName
Type: S3
Name: some-project-pipeline
RoleArn: !GetAtt CodePipelineServicenRole.Arn
Stages:
- Name: Source
Actions:
- Name: GitSource
ActionTypeId:
Category: Source
Owner: ThirdParty
Provider: GitHub
Version: "1"
Configuration:
Owner: !Ref GitHubOwner
Repo: !Ref GitRepo
Branch: !Ref GitBranch
PollForSourceChanges: False
OAuthToken: !Ref GitHubOAuthToken
OutputArtifacts:
- Name: SourceZip
...
- Name: Deploy
Actions:
- Name: DeployPersistantStack
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: "1"
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_NAMED_IAM
RoleArn: !GetAtt CloudFormationExecutionRole.Arn
StackName: some-stack-cfn
TemplatePath: BuildArtifactAsZip::cfn-template.yaml
TemplateConfiguration: BuildArtifactAsZip::dist/config/cloudformation/stack-config.json
InputArtifacts:
- Name: BuildArtifactAsZip
RunOrder: 1
...
CloudFormationExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- cloudformation.amazonaws.com
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
The corner being cut in the code above is easy to find. A CloudFormationExecutionRole having
AdministratorAccess doesn’t apply the least privilege principle. The reason for this shortcut
is because CloudFormation stacks often do a lot of things. Applying the least privilege
rule in this context means a long and well-thought-out policy. For simplicity and speed, I decided
to cut a corner. The trade-off? Allowing the pipeline a tad more access than it should. No big
deal, I could live with that.
A Dangerous Git Backdoor
To give a bit more context: security-wise, our AWS environment is a bit of a fortress: IDP/SSO, AWS WAF, AWS Security Hub, SAST/DAST, EDR and an endless list of tools to close the gates… We made AWS a safe place to go to.
Git repositories often live outside the AWS ecosystem. GitHub, GitLab and Bitbucket are most common to host any code. Are your Git repositories secured as well? If the answer is nah, then here’s some horror. If someone gains access to a Git repository, he could do the following if a pipeline is granted Administrator access:
- Change a DNS Name to re-route traffic to a phishing site
- Change your DeletionPolicy and throw away resources
- Change an instance profile or a security group
- Elevate IAM privileges and create an IAM user for himself
- I think that’s already enough…
Here’s a visualization of the Git backdoor
Asking the same question again: can I live with Administrator access to my build pipeline? The answer is the opposite and it’s a big no-no. I’m feeling lucky I got this epiphany before things turned into a real issue.
How to shut the door?
The best way to close this security hole is to apply the least privilege rule to your pipeline. The downside of this approach is the introduction of extra work. Whenever someone changes a pipeline that incurs security context changes, they must update the pipeline’s policy first. Be prepared for people to be upset by this. Besides, it’s crucial to separate the concern for updating the pipeline policy and disconnect it from Git. Separating the responsibilities by disconnecting the pipeline from Git means it will require two different exploits to gain the same level of access, making changes for an exploit much smaller.
If you are in the situation where your pipeline policy is too relaxed, you could consider starting with an intermediate solution. Start denying all IAM and Route 53 access and/or deny delete access for all resources. Although this is more secure, this approach has downsides. So, you only buy a bit more time to apply the least privilege principle.
Bonus Warning: Self-Mutating Pipelines
For clarity: with self-mutating pipelines I mean pipelines having a stage to update
themselves. Although I was never a fan of pipelines maintaining themselves, in this case, be extra
warned. Having such a pipeline listening to Git means that the Pipeline policy itself can be
changed by a git push.
Even if your self-mutating pipeline doesn’t have administrator access, this is easily changed for self-mutating pipelines using Git access. For that reason, remove self-updating logic from a pipeline. You can use makefiles that require client authentication in order to separate concerns of executing and updating pipelines.
Enjoy and until next time.