Best practices for patching and deprecating Docker images

Intro

One of the biggest benefits of Docker images is their immutability. Once they’re built, they don’t change. Built once, would work forever… That’s how nightmares of security guys starts 🤣

We have then two contradictory concepts:

flowchart LR
    id1(Keep it stable) <---> id2(Keep is up to date and secure)

For day to day work, usually first concept wins. You want your builds stable and try to avoid tempting distractions of upgrading log4j to latest version… Who knows what might break. That’s fine, makes sense.

When is then the good moment to upgrade?

It usually happens once a year, maybe once a 3 years, maybe when we need feature or want to add Java 21 to our CV. Generally speaking “it’s a special event”. For some developers it’s once in a lifetime experience. Taking into account how long they would stay in a company, they might not be there next time it happens.

There are obvious security risks of not performing updates, we saw it multiple times:

Question is: How to maintain updates, allowing developer to work?

Let my share how we do it.

Deprecate before deleting

By deprecation we mean: mark as outdated, unsupported, smelly and crappy. It’s working still (on best effort) but will be gone soon. We usually point what should be used instead.

Deprecation is a process which we perform for old Docker images, that we would like to drop or outdated tools which we don’t have resources to support anymore.

How to deprecate Docker image?

It’s simple! Just drop something like this to your docker file:

...

ONBUILD RUN echo << EOF

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣠⣤⣤⣤⣤⣀⣀⣀⣀⣀⣀⣀⣀⣠⣀⣤⣤⣤⣤⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⣶⠟⠉⠉⠉⠋⠋⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠛⢙⠙⢿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⠟⠁⣠⣾⠟⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠛⠻⣷⡄⠙⢿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣼⡟⠋⣠⣾⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢻⣦⡀⠙⢿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⡿⠋⣠⣾⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⠿⣦⡀⠙⢿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⡿⠋⣠⣾⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣦⡀⠙⣿⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢰⣿⠋⣠⣾⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣦⡈⢹⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⣿⡇⠀⠀⢀⣠⣤⣤⡀⠀⢠⣤⣤⣤⣤⣤⣤⣤⠀⠀⢀⣤⣤⣄⡀⠀⠀⣠⣤⣤⣤⣤⣀⠀⠀⠀⢹⡇⢰⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⢿⡇⠀⣴⡟⢻⣭⣍⢻⣷⣸⣯⣍⡍⢫⣍⣉⣿⡄⣾⠟⣿⣩⣉⠻⣦⠀⣿⡏⢹⣏⣉⡛⣿⣄⠀⢸⡇⠨⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⢸⡇⠀⣿⠀⣿⡉⣿⣷⣿⡟⠛⢻⣟⢘⣿⠉⠉⢸⡏⢠⣿⠙⣿⠀⣿⡇⣿⡇⢸⡟⢻⡇⠘⣿⠀⢸⡇⢨⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⢸⡇⠀⣿⣆⠸⣷⡉⠉⠉⠁⠀⢸⡯⠰⣿⠀⠀⢸⡇⠸⣿⠀⣿⠀⣿⡇⣿⡇⢸⣇⢸⣧⠀⣿⠀⢸⡇⠐⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⢸⡇⠀⠈⢻⣦⡈⠻⣦⡀⠀⠀⢸⣇⠀⣿⠀⠀⢸⡇⠈⣿⠀⣿⠀⣿⡇⣿⣇⠸⠿⡿⣣⣼⡟⠀⢸⡇⠀⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⢸⡇⢁⣀⣀⣈⢿⣆⠘⣿⡆⠀⢸⡗⠀⣿⠀⠀⢸⡇⠀⣿⠀⣿⠀⣿⡇⣿⡟⢻⡿⠾⠿⠋⠀⠀⢸⡇⠀⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⢸⡇⠀⣿⠙⣿⣁⣿⠇⣿⡇⠀⢸⡧⠀⣿⠀⠀⢸⣧⠐⣿⣀⣿⠀⣿⡇⣿⡇⢸⣿⠀⠀⠀⠀⠀⢸⡇⠀⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⢸⡇⠀⢿⣧⣙⡛⣋⣴⣿⠁⠀⢸⣿⣰⣿⠀⠀⠀⢿⣆⣙⢛⣋⣴⡟⠁⣿⣇⣸⣿⠀⠀⠀⠀⠀⢸⡇⠀⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⣸⡇⠀⠀⠉⠛⠛⠛⠉⠀⠀⠀⠈⠛⠛⠛⠀⠀⠀⠀⠙⠛⠛⠛⠉⠀⠀⠙⠛⠛⠋⠀⠀⠀⠀⠀⣸⡇⠀⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣄⠘⢿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⡿⠁⣰⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣦⡀⠙⢷⣆⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⡿⠋⣠⣾⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣦⡀⠹⢿⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⡿⠋⣠⣾⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣦⡀⠻⣷⣆⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⡿⠋⣠⣾⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣦⡈⠻⣷⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣴⡿⠋⣠⣾⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣦⣀⣀⣀⣀⣀⣈⣉⣉⣀⣀⣀⣀⣁⣁⣀⣀⣀⣀⣀⣠⣼⠟⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠛⠛⠛⠛⠛⠛⣿⡟⠛⠛⠛⢛⣿⡟⠛⠛⠛⠛⠛⠛⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⡇⠀⠀⣼⡆⢹⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⡅⠀⠀⣿⡇⢸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⡂⠀⠀⣿⡇⢸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⡅⠀⠀⣿⡇⢸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⡇⠀⠀⣿⡇⢸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⡇⠀⠀⣿⡇⢸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⡅⠀⠀⠀⠀⢸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢿⣿⠶⠶⠶⠶⢾⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤
⣘⠀⠀⣙⡋⣛⣋⢋⢛⣛⣟⡏⣛⣛⣙⣻⡟⡋⢛⢉⣿⡄⣀⣀⢀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⢀⢀⣀⡀⠀⠀⢀⢀⣀⡀⣈⣉⣉⣉⣉⣉⣉⣙⣙⣙⣛⣉⣉⣙

This Docker image is deprecated and will be soon deleted!

Please use XYZ instead.
In case of issues, contact:
  #best-team-ever
  best-team-ever@corpo.net

EOF

This command will print our clear message when people will try to use it.

Make upgrades predictable

We seriously upgrade our base images every quarter. By seriously, I mean we’re touching everything possible to upgrade. We pin (hardcode) versions for all the changes, to ensure stable rebuilds. That’s usually a week of work for the team and then we drop a bomb by releasing those updates…

graph TB
    A(Upgrade everything that can be upgraded) --> B(Introduce new images)
    B --> C(Deprecate outdated)
    C --> D(Delete previously deprecated)
    D -- repeat
quarterly --> A

Make upgrades visible

Before we push upgrades or even before we start working on them, we’re performing a quick research on what we might be delivering:

• check new versions available (new Java, new Python, etc),
• check what we will drop (deprecated last time),
• check what we might change breaking contract and causing potential configuration issues.

Usually it takes around 30~60 minutes to quickly scan what we planned to do last quarter, what will change right now. We use Semantic Versioning¹, so we focus here on MAJOR and MINOR changes.

We send a quick mail with the summary of changes to all the IT people. Week or two later changes are deployed. It’s not much time you might say, that’s true, but it’s not our first notification. After we release upgrade we send more detailed mail with:

• all the changes that happen,
• all the deprecations (more on that later),
• plans for bigger changes for next quarter.

sequenceDiagram
    Sprint start ->> Quarterly patching: Send email
    Note right of Sprint start: at beginning of the sprint,
short reminder,
raw plans
    Quarterly patching ->> Sprint end: Send mail again
    Note left of Sprint end: detailed
new,
updated,
deprecated,
dropped

This makes big difference, because dev teams know what will happen in 3 months from now. They can start prepare and plan time for the change. They can own it!

Is it enough to upgrade quarterly?

No, it’s not.That’s a bare minimum. If you do just that, you will be vulnerable just a few weeks after the Quarterly patching.

We try to limit, big breaking changes and upgrades to those quarterly events, but we still perform smaller upgrades between those big events.

It happen twofold:

1. We rebuild our base images monthly.

   Each rebuild will update OS system packages and install recent updates. This makes images bigger, but also us safer and is as simple as adding:

   RUN apt update && apt dist-upgrade -y
   

   to our top level base images.

   Those are unattended and completely automated events. No big changes happen here, so they rarely cause trouble.

2. We usually work on Docker images between upgrades.

   Adding new features or fixing bugs. Those are usually MINOR or PATCH level changes, so they don’t cause much fuss.

   Each our change, triggers action from pt. 1 - upgrading OS packages.

graph LR
    A((Monthly rebuild\nor on-demand))
      --> B(Upgrade OS packages)
      --> C((Push))

Ask devs to use latest Docker image tag by default

Christ, latest! Seriously?

Yes. We maintain our own base images and we encourage developers to rely on the latest tag by default. They can “fallback” to the tagged version if they face issues. This usually follows up with a ticket to check what’s wrong and get back “to the bright side” in next sprint.

This way they use recent images most of the time and when they’re not - it’s their responsibility to get back on the latest.

We (as a team) do not provide help/support if someone use non latest image version.

Ensure latest tags are stable - test Docker images!

We prepared a testing framework that allows us to test Docker images after they’re build. We used for that Pytest  and Testinfra  . Together with some bash in Makefile, we’re listing all the images and checking if they behave how we expect, if they start, run as non-root, etc.

Testinfra allows to easily check any condition  . Pytest spins Docker image per set of tests. For around 200 images that we maintain, we run more than 1500 tests. Since we started doing this, it’s really hard to break basic functionality of images. This allows us to push harder with confidence that latest versions are stable.

Sum it up

What was the impact of implementation of this process?

There are many positive aspects of it:

1. Scheduled upgrades: Changes are communicated in advance, reducing the need for immediate requests.
2. Stability through testing: Rigorous testing minimizes the risk of disruptions during upgrades.
3. Informed stakeholders: Developers and Product Owners receive early notifications, enabling better planning.
4. Efficient deprecation process: Notices allow for timely reactions and signal the seriousness of image changes.
5. Reduced change requests: Teams anticipate updates within the quarterly cycle, reducing the frequency of requests.
6. Seamless bug fixes: Bug fixes and new features are implemented between major upgrades without causing disruptions.
7. Clear deletion timeline: Deprecated images are scheduled for deletion in the following quarter, accelerating legacy removal.
8. Collaborative issue resolution: Complaints prompt discussions and options for resolution, fostering collaboration.
9. Flexibility in usage: Teams have the autonomy to choose whether to stay on older image versions if needed.

Updates have to be delivered frequently - it have to be boring process, that every body expect. It is for us 🤓

Enjoyed?