DevOps
Cluture & Best Practises
All processes and artifacts can be found at this url https://openpracticelibrary.com
The basis of this document is to elaborate on the course DO500 offered by Redhat. It has valuable information/processes that can be implemented and fed directly into our Agile/Scrum framework.
Overview
The underlying process is best described in the diagram below
The process is fairly straight forward. There are 2 main loops with an options pivot separating the loops. These loops are namely Discovery and Delivery, the options pivot allows us then to move from one loop to the other.
The process is started by defining and setting the foundation.
The foundation is split in 2 parts :
- Culture & Collaboration
- Technical
The Discovery loop defines the WHY and sets OUTCOMES, while the Delivery loop sets our DELIVERY with MEASURE&LEARN processes
Culture & Collaboration
The whole process is started by defining the following :
Social Contract (this list is by no means exhaustive - add what you think would benefit the team). When everyone agrees to the social contract , each member signs it.
- No idea is useless
- Best idea wins
- Having fun
- Define working hours
- No black magic
- Respect each others concerns
- No stupid questions
- Standup every morning at 9 (for this session only)
Visualize Everything
All artifacts/processes are visualized
This is limiting in a remotely distributed team. It therefore makes sense to have a week set aside for a full face to face co-located session
It also means that real estate is needed and that we use quality post-it's for all visuals
Items include :
- Parking lot
- Target Outcomes
- Impact map
- Event storming
- Design sprint - Lean UI/UX etc
- Burndown chart
- Social contract
- Target outcomes
- Definition of done
- Hopes/Fears
- Priority sliders
- Overview architecture board (e.g. CI/CD, Openshift, UAT, PROD,product poc, etc)
- Team sentiment
- Real time retro
- Kanban board
- Value mapping/slicing
- Relative sizing
- Pairing and mobbing
Definition of Done
- The team decides what the definition of done is for the session (this is not the same as the dev teams DOD)
Kanban Board
- Post-its are moved from backlog (defined by the DevOps facilitator) to in progress to done
Parking Lot
- When the team can't agree or there are to many question/assumptions or not enough SME info, the item gets put on the parking lot and is addressed after standup each day
Relative Sizing
- Using a fibonacci sequence to size user stories
Celebrate Failures
- We learn from our mistakes - its not a bad thing
Pairing and Mobbing
- Pair up with another developer to understand and code a service/s
- Mobbing is really programming in a group, i.e one person codes for 7 minutes and then the next person takes over - all members are involved from junior to senior
Burndown Chart
- Updated at standup every morning
Team Sentiment
- At any time in the process the team members can add green (OK post-it's) or red (NOTOK post-it's) to indicate the overall team sentiment. This also gets addressed each day after standup.
Stop the World
- At any time any team member can use the "Stop the World" card. This means that everyone stops what they are doing and address the concerns of the person who used the card.
Realtime Retro
- At any time any team member can add a post-it to the "realtime retro" board in the form of keep doing, stop, or start. The items are reviewed after every standup
Technical
In this section we define our automated processes namely :
- CICD Pipeline Architetcure
- Continuous Integration
- Continuous Deployment
- Continuous Delivery
- Everything as code
- Test Driven Development
- Trunk Driven Development
- Contract first (API design)
- Test Automation
- Visible Architecture (eg. CICD, product poc etc)
Reference Architecture (Based on the DO500 course)
The Big Picture is our emerging architecture; starting with an empty cluster we populate it with projects and some CI/CD tooling.
Implemented Solution
The architecture is visible to all and if we have a process already in place the pipeline dashboard is also made visible to all. The pipeline can be used now for POC's and development etc. In our CICD pipeline we have made use of the following technologies - all hosted on Openshift. Technologies should not be enforced, it's the process we're after.
- ArgoCD - Tekton (openshift pipelines)
- SonarQube and Postgresql
- Gitea and MariaDB (golang opensource version of Git)
- Kaniko (from GoogleContainerTools) to build Linux containers/images - instead of s21 build
Once a merge has been made to the trunk the pipeline is triggered
The pipeline process is as follows (any failure will stop the process and is visible via the dashboard) :
This applies to both nodejs/golang application development
- Checkout the code
- Compile the code (golang)
- Run unit tests
- Execute sonar scanner
- Build and push Linux image (Kaniko job)
- Deploy to UAT, PRE-PROD or PROD (Openshift CLI)
- Execute E2E testing
Once the process has been succesfully pushed to PROD - a MANUAL merge to stash (Bitbucket) is performed
The process is straightforward. The developer checks out code from the master branch (gitea) creates "push requests" against master (Trunk). All changes are made to this branch. Once the developer is satisfied that the changes made has :
- been tested locally and working,
- met the code linting requirements
- met the code analysis requirement (code coverage, commenting, critical and major bugs/issues, code smell and vulnerability/security issues)
- been peer reviewed
The developer can then merge to master (gitea) , this then triggers the pipeline.
The reasons for using Trunk Driven Development (as stated in the book "ACCELERATE - Gene Kim et al.") is as follows
- Small batches (with statistical data proving this is a better process)
- High Performance teams keep branches short lived - with frequent integrations into master
- Each change triggers a build process that includes running unit tests
- If any step fails - the developers fix it immediately
- It avoids merge hell (with long lived feature branches)
Implementation (Discovery Loop - WHY & OUTCOMES) Once the foundation has been agreed upon the process kicks off using the following processes in the specified order - Target Outcomes - Start at the End - Event Storming (DDD) - (time box to 2 days) - Metrics based process mapping - Impact Mapping - Priority Sliders - Value Slicing The team can opt to pivot into the delivery loop or continue in the discovery loop Implementation (Options Pivot) From the event storming each command (discussed later) represents a user story. The output of the impact mapping, priority sliders and value slicing can then be fed into a prioritized product backlog. - Design Sprint (if needed) - Lean UI/UX - User Testing - Empathy Mapping - Backlog Refinement - Implementation (Delivery Loop - DELIVER, MEASURE&LEARN ) We can now follow our standard scrum ceremonies : - Sprint Planning - Daily Standup - Show Case (review) - Retrospective The team can opt out into the Discovery loop or remain in the Delivery loop In this loop we also adopt the following strategies : - Contract First Development (OpenAPI) - Canary Deployments - Blue/Green Deployments - Event Storming (Domain Driven Design) The event storming process is more complicated, in this section we try to define it more clearly.
![]()
The process must adhere to the color codes and post-it styles.
This forms part of the ubiquitous language in DDD. The following is a definition of each item
- User - the specific user acting on a command
- Command - action started by an actor/user (basis for user story)
- Event - is in the form of subject verb (and usually occurs in the past) - i.e - payment made
- System - the underlying system that supports the command/event i.e - PayPal for payments
- Aggregate - core or base model (geographical rock is the basis for defining an aggregate)
- Read Model - information needed to make a decision
- Policy&Procedure - i.e what policies are needed to make a decision (i.e roles and permissions)
- Questions - Pain points , what/why/how/where
- Assumptions - what assumptions are made that support the decision/command/event
- Sub-Process - a dependency on a sub-process (i.e verify credit card)
- Screen Layout - a basic wireframe indicating the need to capture data - may lead to design sprint
The process is as follows (e.g. we are designing for a Payment)
- We start by defining an event i.e payment made (subject and verb) (orange post-it)
- We place a command that is the inverse of the event i.e make payment. (lightblue post-it)
- Place the user/actor that creates the command (i.e buyer - yellow post-it)
- Add an Aggregate - Payments (lightyellow rectangle post-it)
- We can add a System post-it (lightred post-it rectangle) - i.e PayPal
- We can add assumption/s or question/s if we are not sure, i.e. do we login ?, do we validate credit details ? etc (bright pink)
- We can also add a policy i.e. saying that the connection must be secure (light pink)
- Adding a Read Model - we can say that we need to capture credit card details and list the fields (green)
- For this process we need a screen to capture the user info - (green rectangle)
Activity Breakdown
We set a list of activities over a 5 day intense engagement (this can be tailored if the project is not dependant on data) as well as the list of required attendies.
Required Attendies
- Product Owner
- Product Manager
- Systems Architect
- Data Architect
- InfoSec
- IT
- Legal
- Finance
- Developer/s (SME)
- UI/UX Designer/s
- UI/UX Developer/s
The breakdown of activities is as follows :
Day 1
All visuall boards should be setup by the co-ordinator/fascilitator
These include
- Parking Lot
- Team Sentiment
- Burndown Chart
- Kandban (backlog, in progress, done)
- Social Contract (place holder)
- Definition of Done (place holder)
- Iteration Overview Diagram
- Architectural Diagram (System & CICD)
- Activities
- Welcome and Introduction
- Agree on Social Contract (Team)
- Agree on Definition Of Done (Team)
- Discusion of foundations (Culture & Collaboration)
- Parking Lot
- Celebrating Failures
- Team Sentiment
- Pairing and Mobbing
- Burndown Chart
- Stop the World
- Relative Sizing
- Visualization of Work
- Realtime Retro
- Discussion of foundations (Technical)
- Test Driven Development
- Continious Integration
- Continious Deployment
- Continious Delivery
- Test Automation
- Deployment Strategies
- Everything as Code
- Update Burndown chart
Day 2
- Daily Standup
- Review Live Retro
- Review Parking Lot
- Update Burndown Chart
- Other issues
- Start at the End
- Metrics Based Process Mapping
- Event Storming
- CICD process update
Day 3
- Daily Standup
- Review Live Retro
- Review Parking Lot
- Update Burndown Chart
- Other issues
- Event Storming continued (optional)
- POC Code development (optional)
- CICD process update
Day 4
- Daily Standup
- Review Live Retro
- Review Parking Lot
- Update Burndown Chart
- Other issues
- Event Storming continued (optional)
- POC Code development (optional)
- CICD process update
- Impact Mapping
- Value Slicing
- Priority Sliders
- Design Sprint (identify if needed)
- Lean UI/UX
- Empathy Mapping
- User Testing etc
Day 5
- Daily Standup
- Review Live Retro
- Review Parking Lot
- Update Burndown Chart
- Other issues
- Options Pivot ?
- Prioritise Backlog
- CICD process update
- Summary / Feedback
The process is fairly straight forward. There are 2 main loops with an options pivot separating the loops. These loops are namely Discovery and Delivery, the options pivot allows us then to move from one loop to the other.
The process is started by defining and setting the foundation.
The foundation is split in 2 parts :
- Culture & Collaboration
- Technical
The Discovery loop defines the WHY and sets OUTCOMES, while the Delivery loop sets our DELIVERY with MEASURE&LEARN processes
Culture & Collaboration
The whole process is started by defining the following :
Social Contract (this list is by no means exhaustive - add what you think would benefit the team). When everyone agrees to the social contract , each member signs it.
- No idea is useless
- Best idea wins
- Having fun
- Define working hours
- No black magic
- Respect each others concerns
- No stupid questions
- Standup every morning at 9 (for this session only)
Visualize Everything
All artifacts/processes are visualized
This is limiting in a remotely distributed team. It therefore makes sense to have a week set aside for a full face to face co-located session
It also means that real estate is needed and that we use quality post-it's for all visuals
Items include :
- Parking lot
- Target Outcomes
- Impact map
- Event storming
- Design sprint - Lean UI/UX etc
- Burndown chart
- Social contract
- Target outcomes
- Definition of done
- Hopes/Fears
- Priority sliders
- Overview architecture board (e.g. CI/CD, Openshift, UAT, PROD,product poc, etc)
- Team sentiment
- Real time retro
- Kanban board
- Value mapping/slicing
- Relative sizing
- Pairing and mobbing
Definition of Done
- The team decides what the definition of done is for the session (this is not the same as the dev teams DOD)
Kanban Board
- Post-its are moved from backlog (defined by the DevOps facilitator) to in progress to done
Parking Lot
- When the team can't agree or there are to many question/assumptions or not enough SME info, the item gets put on the parking lot and is addressed after standup each day
Relative Sizing
- Using a fibonacci sequence to size user stories
Celebrate Failures
- We learn from our mistakes - its not a bad thing
Pairing and Mobbing
- Pair up with another developer to understand and code a service/s
- Mobbing is really programming in a group, i.e one person codes for 7 minutes and then the next person takes over - all members are involved from junior to senior
Burndown Chart
- Updated at standup every morning
Team Sentiment
- At any time in the process the team members can add green (OK post-it's) or red (NOTOK post-it's) to indicate the overall team sentiment. This also gets addressed each day after standup.
Stop the World
- At any time any team member can use the "Stop the World" card. This means that everyone stops what they are doing and address the concerns of the person who used the card.
Realtime Retro
- At any time any team member can add a post-it to the "realtime retro" board in the form of keep doing, stop, or start. The items are reviewed after every standup
Technical
In this section we define our automated processes namely :
- CICD Pipeline Architetcure
- Continuous Integration
- Continuous Deployment
- Continuous Delivery
- Everything as code
- Test Driven Development
- Trunk Driven Development
- Contract first (API design)
- Test Automation
- Visible Architecture (eg. CICD, product poc etc)
Reference Architecture (Based on the DO500 course)
The Big Picture is our emerging architecture; starting with an empty cluster we populate it with projects and some CI/CD tooling.
Implemented Solution
The architecture is visible to all and if we have a process already in place the pipeline dashboard is also made visible to all. The pipeline can be used now for POC's and development etc. In our CICD pipeline we have made use of the following technologies - all hosted on Openshift. Technologies should not be enforced, it's the process we're after.
- ArgoCD - Tekton (openshift pipelines)
- SonarQube and Postgresql
- Gitea and MariaDB (golang opensource version of Git)
- Kaniko (from GoogleContainerTools) to build Linux containers/images - instead of s21 build
Once a merge has been made to the trunk the pipeline is triggered
The pipeline process is as follows (any failure will stop the process and is visible via the dashboard) :
This applies to both nodejs/golang application development
- Checkout the code
- Compile the code (golang)
- Run unit tests
- Execute sonar scanner
- Build and push Linux image (Kaniko job)
- Deploy to UAT, PRE-PROD or PROD (Openshift CLI)
- Execute E2E testing
Once the process has been succesfully pushed to PROD - a MANUAL merge to stash (Bitbucket) is performed
The process is straightforward. The developer checks out code from the master branch (gitea) creates "push requests" against master (Trunk). All changes are made to this branch. Once the developer is satisfied that the changes made has :
- been tested locally and working,
- met the code linting requirements
- met the code analysis requirement (code coverage, commenting, critical and major bugs/issues, code smell and vulnerability/security issues)
- been peer reviewed
The developer can then merge to master (gitea) , this then triggers the pipeline.
The reasons for using Trunk Driven Development (as stated in the book "ACCELERATE - Gene Kim et al.") is as follows
- Small batches (with statistical data proving this is a better process)
- High Performance teams keep branches short lived - with frequent integrations into master
- Each change triggers a build process that includes running unit tests
- If any step fails - the developers fix it immediately
- It avoids merge hell (with long lived feature branches)Implementation (Discovery Loop - WHY & OUTCOMES) Once the foundation has been agreed upon the process kicks off using the following processes in the specified order - Target Outcomes - Start at the End - Event Storming (DDD) - (time box to 2 days) - Metrics based process mapping - Impact Mapping - Priority Sliders - Value Slicing The team can opt to pivot into the delivery loop or continue in the discovery loop Implementation (Options Pivot) From the event storming each command (discussed later) represents a user story. The output of the impact mapping, priority sliders and value slicing can then be fed into a prioritized product backlog. - Design Sprint (if needed) - Lean UI/UX - User Testing - Empathy Mapping - Backlog Refinement - Implementation (Delivery Loop - DELIVER, MEASURE&LEARN ) We can now follow our standard scrum ceremonies : - Sprint Planning - Daily Standup - Show Case (review) - Retrospective The team can opt out into the Discovery loop or remain in the Delivery loop In this loop we also adopt the following strategies : - Contract First Development (OpenAPI) - Canary Deployments - Blue/Green Deployments - Event Storming (Domain Driven Design) The event storming process is more complicated, in this section we try to define it more clearly.