This article describes the Puppet infrastructure group’s process of vetting Puppet Enterprise patch releases for the current Long Term Support (LTS) version of Puppet Enterprise (2021.7.x) and timeline expectations for having the release available to PE consumer groups, as well as methods for testing against patch releases prior to primary PE servers being updated.
Puppet infrastructure group’s role in testing patch releases.
When a patch update is made available, the release notes are reviewed. PE component release notes are also reviewed. If nothing conspicuous stands out, then the team proceeds with updating a PE testing server. Then a CentOS 7 Linux testing node is tested against the PE testing server; the agent is updated, an environment is deployed (in this case ITS-EI environment), and catalog compilation runs. We then review reports and server logs for anything unexpected. If there’s anything unexpected, we work toward a resolution or workaround.
There are many tools for the PE platform (PE Client Tools, PDK - Puppet Development Kit) and components (Puppet, Puppet Server, Facter, Hiera, and PuppetDB), PE platform server components (r10k, Bolt, PostgreSQL, Java, Nginx, Ruby, OpenSSL), and many Puppet agent versions (MS Windows, macOS, AIX, Amazon Linux, Debian, EL; CentOS, Oracle Linux, RHEL, Scientific Linux., Fedora, Suse, Ubuntu, Amazon Linux). Unfortunately, we do not have the time or resources to test every scenario for minor releases, so we strongly encourage consumers to do their own testing on the components that impact their operations during the week allotted for consumer testing (see timeline below).
Timeline for applying patch updates.
- Puppet (the vendor) notifies customer (U of Iowa) of new minor patch release (week 0)
- Patch update applied to PE testing server (week 1)
- Puppet infrastructure testing against PE testing server (week 2)
- Notify PE consumer groups of patch update (week 3)
- Week allotted for testing by PE consumer groups (week 4)
- Apply patch to first group of PE consumer servers (week 5)
- Apply patch to second group of PE consumer servers (week 6)
It is possible that updates are applied quicker, but this amount of time allows people to take vacation, work on higher priority items, and/or fix any potential issues with the update. Updates may be expedited at the request of ISPO or OneIT leadership in cases where there is a critical impact to existing infrastructure.
What if a PE patch release breaks something? (node runs fail, code/agent is deprecated, or tools don’t work as expected)
Reviewing release notes can avoid some issues but with all the components, tools, and mixture of diverse Puppet code between consumer groups on campus, unforeseen issues can come up. The Puppet infrastructure group can help troubleshoot issues or open support tickets with Puppet. If there’s quick fix to code, we’d try that first. If there’s no immediate fix, reverting the consumer group’s primary PE server from snapshot is the quickest solution to get back up and running. PE server snapshot rollback can be done within an hour. If tools don’t work as expected with a version of PE, it’s usually the latest version of that tool, and the expected workaround is to use the previous version of that tool. Typically, the version you were using before the upgrade.
Consumers testing code or nodes prior to a Puppet Enterprise minor patch release update.
Consumer groups can test node(s) or code against a PE2021.7.x test server by requesting help from the Puppet infrastructure group at its-puppet-infra@iowa.uiowa.edu. The PE2021.7.x testing server would be setup for the user(s), node(s), and environment(s) temporarily and be allowed to test on that server for a day or two. If there’s contention for testing and not enough resources to complete desired testing, then the timeline for applying updates may need to be lengthened.
If consumers have requests for testing other features ahead of time, or specific use case testing, these can be discussed by emailing the Puppet infrastructure team at its-puppet-infra@iowa.uiowa.edu.
PE consumers may choose to do their own testing without a PE2021.7.x testing server. See Testing Puppet Code & Modules.
This update process, timeline, and testing procedures differ from a PE LTS release update, which happens every 2 years. See Puppet Enterprise Version Support and Release Lifecycle, and in that case, a new PE server is built for the consumer group with 6 months of testing and node migrations.