# tl;dr
These slides I've used for a private training on secret management for Java developers. I am publishing them as a simple reference for basic AppSec concepts and as a template for lazy penetration testers (like me) to create their own on-demand training materials for compliance or other reasons.
---
# secure secret management (from an offensive PoV)
---
# introduction
```bash
export AUTH_TOKEN="Welc0me+yMTBjMTIwZmNkNDEzNDNmZDJlYjg3MzRmYzk1M2RlZTA0ZWQzMTgzMGRhNzAxNzNjZGIzZTU2MWJlN2JiMDEzOGFhY2Q1ZDI3N2EzY2ZlNmNiNzE5NGYxY2ViNzUyOGIwMzI3ZjQ4OTc5ZDQ5OTBhZDJhY2I0ZTU"
```
---
## training approach
*operators always wear gloves for protection*
*screenshot redacted*
---
## what is a secret? 🗝
- passwords
- persistent tokens
- private keys and HMAC secrets
---
## threat model range
- external attacker
- authenticated access to an application
- RCE in a docker container
- access to a single server
- access to a developer's workstation
---
## furher reading, theory and guidelines
https://owaspsamm.org/model/implementation/secure-deployment/stream-b/
https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Secrets_Management_Cheat_Sheet.md
*and numerous tooling documentation*
---
## labs and materials used throughout training
- OWASP WrongSecrets
- anonymized excerpts from penetration test reports
- simple custom setups to illustrate particular points
---
# Docker containers
> [!NOTE]
> Container breakouts and other misconfigurations aside secret management are out-of-scope of this training.
```bash
root@container # cat .env
export AWSJWT="eY..."
```
---
## containers: threat model
- external attacker
- **authenticated access to an application** (for some cases)
- **RCE in a docker container**
- **access to a single server**
- access to a developer's workstation
---
## secrets in files
*the most common secret management issue*
```Dockerfile
COPY --chown=container /home/user /home/container
```
```bash
container@1cd133db52a0:/$ cd
container@1cd133db52a0:~$ ls .ssh/
id_rsa id_rsa.pub known_hosts
```
---
### secrets in files: COPY and ADD from developer workstations
- build scripts
- `git` secrets
- curl commands
- credential caches
- `env` files
---
### secrets in files: `.xyz_history`
- `bash`, `zsh`, `fish`, `sh`, and other shells
- `.mysql_histfile`, `psql` history files, and other dbs
- command-line editor or IDE caches
- can be **carried over** when creating a container
---
### secrets in files: simple demo
*refer to: `casts/docker-history.cast`*
---
### secrets in files: other source examples
- SSH keys
- local account passwords
- local service secrets
---
## secrets in environment variables
> [!NOTE] Note that environment variables can be easily leaked by the application itself, not requiring any access to the container.
```Docker
ENV AWSTOKEN=$secretToken
```
```bash
$ export AUTHT="1122334455667788"
```
---
### secrets in env: common cases
- API keys (`curl -H "Authentication: $KEY" ...`)
- AWS/Azure/GCP access tokens
- build environment secrets
- *even passwords (sometimes)*
---
### secrets in env: process environments
- not only `env` in bash
```bash
find /proc -name environ -exec xargs -0 -L1 -a {} \\; 2>/dev/null | sort --uniq
```
---
### secrets in env: demo
*refer to: `casts/docker-env.cast`*
---
## secrets in memory
> [!NOTE] It is possible to erase secrets from the memory entirely, akin to how SSH does it, but doing so it out-of-scope for this training.
- hard to avoid exposure
- the least-privilege principle is **very** important to apply
---
### host -> container lateral movement case
- sometimes the attacker has access to the host running the container
- monitoring containers may discourage the attacker to run `docker exec`
- it is somewhat beneficial to keep secrets in non-swappable memory
---
## secrets in `build` arguments, `history` and docker layers
- temporary files (`docker image inspect myimg | jq '.[0].RootFS.Layers'` and viewing the diff)
- logging sensitive data
- secrets in `Dockerfile`
---
### secrets in dockerd: argument demo
*refer to: casts/docker-arg.cast*
---
### automated scanners (`dockle`/`trivy`/`secretscanner`)
> [!Note] Most of those can be easily integrated into a CI/CD pipeline.
- regex rules for file contents or paths
- regex rules for libraries coupled with CVE
- "smarter" secret detection
---
### scanners: `dockle` demo
*refer to: casts/dockle.cast*
---
### scanners: `secretscanner` demo
*refer to: casts/secretscanner.cast*
---
### scanners: `trivy` demo
*scans libraries, but wors at scanning secrets*
*refer to: casts/trivy.cast*
---
## other external secret management solutions (e.g. `Hashicorp Vault`/`docker secrets`)
- benefits of RBAC
- alerts and monitoring
- application integration
- "JIT" secret use
---
# `bash`/`zsh` scripts
```bash
if [[ -v KEY ]]; then
export KEY="9e61f1c8210c120fcd41343fd2eb8734"
fi
curl -H "Authorization: admin:pass" -H "X-Key: $KEY" http://localhost:8001
```
---
## scripts: threat model
- external attacker
- authenticated access to an application
- **RCE in a docker container**
- **access to a single server**
- **access to a developer's workstation**
---
## `export` files
```bash
export KEY1=...
export KEY2=...
export TOKEN=...
```
- insecure, but still widely used
- almost the worst case scenario for script secret management
---
## auditing shell scripts
> [!NOTE] Could be integrated into a CI/CD pipeline.
- custom auditing for shell scripts in general
- filtering for `export`, `sshpass` and other hard-coded secrets
- regex-based secret detection akin to `trufflehog`
---
## scripts: `keyctl`/`pass`/`gpg`
- decryption of passwords in runtime (with its own disadvantages)
- secure storage of secrets in kernel memory
```j
APIKEY=$(pass Keys/apikey) ./run
vs
APIKEY=$(< ~/.local/share/apikey) ./run
```
---
## scripts: external audited store
- auditing and alerts
- could provide a stable IoC
```j
SECRET=$(vault kv get -field foo secret/mysecret)
```
---
# logging
```bash
$ cat /data/logs/application.log | grep -i token | wc -l
37
```
---
## logging: Docker, scripts and build processes
> [!NOTE] An example of this is also shown above.
- may contain heaps of environment information, leaking secrets in the process
- also may include a full cmdline of failed commands
```bash
docker logs <containerID>
```
---
## logging: application logs
> [!sidenote] Also applies to error handling, although out-of-scope for this training.
- applications often expose relevant environment details or implementation information in logs
- implementing logging and implementing logging sync are often done by different people
- logs are often excluded from the threat model
---
## logging: auditing logs
- regexes on test environments
- manual review (if there is some spare time)
- red team (if they get to them)
---
# git secret leaks
```
$ git clone
[email protected]:cicd/environment.sh
```
---
## git: threat model
- **external attacker**
- **authenticated access to an application**
- **RCE in a docker container**
- **access to a single server**
- **access to a developer's workstation**
---
## hardcoded secrets
Not just `String pass = "N0OneW1llG3tTh1s";`:
- secrets in automated tests and deployment configuration
- secrets in sample front-end code
- hard-coded signatures and HMAC keys
- testing credentials that "bypass" security checks
---
## `.git` directories
Access to the private Gitlab is not the only way attackers get `git` access.
- exposure of a `.git` directory on
- Git repositories used to pull configurations by build agents
---
### `.git` directories: demo
*refer to casts/git-demo.cast*
---
## commits, pulls and branches
- leaked secrets may persist in commit history
- do not only scan the `main` branch
- do not remove secrets by just committing a new version
---
## `trufflehog`
- the standard for automated secret scanning (especially `git`)
[](https://camo.githubusercontent.com/5661f8ac591a9f60fb0faeb9f70dacd0b915e6ab0e2049a68159be1dea3b1382/68747470733a2f2f73746f726167652e676f6f676c65617069732e636f6d2f74727566666c65686f672d7374617469632d736f75726365732f706978656c5f7069672e706e67)
---
### `trufflehog` usage: demo
*refer to casts/trufflehog.cast*
---
### third-party services + git: real-world example
found on `Confluence`:
*screenshot redacted*
*blurring would not help*
---
after `git clone`:
*screenshot redacted*
---
# other sources
---
## public info: threat model
```
where else can you look for secrets?
```
---
## storage buckets, static landing pages and other static content
- can be enumerated, sometimes surprisingly so
- ranges from `.git` and deploy scripts to secrets in static JS
- can contain secrets that are only usable in the application context (e.g. session encryption keys)
---
### landings and static content: example
A landing page deployed automatically may contain this:
*screenshot redacted*
---
### buckets: examples
lots of "lists of shame":
*screenshot redacted*
---
## third-party services
A list of examples *ordered by frequency of exploitation*:
- File sharing solutions **(the worst of them all)**
- Jira/Confluence/other knowledge bases
- Backup solutions *especially self-developed ones*
---
### third-party services: what can happen
This:
*screenshot redacted*
---
# high-level mitigations
> [!note] Most mitigations are high-level here.
- tight secret control
- detectability and logging
- rotating credentials
- MFA
---
# conclusions
- secret management is hard, but doable
- automated scanning *drastically* reduces exposure, because attackers use them too
- auditing secret access is *the* approach to tackling and containing leaks
- practical auditing could be conducted by a developer with minimal preparation
---
# questions
`if you have any`