Introduction

Many developers and organizations embark on their journey with Continuous Integration and Continuous Deployment (CI/CD) full of optimism, yet the real-life experiences of countless developers reveal that the path to mastering CI/CD is fraught with complications, unexpected setbacks, and sometimes overwhelming obstacles. As we strip away the buzz and the lofty claims about CI/CD revolutionizing development, we often find developers entangled in tedious manual setups, scrambling to manage breakdowns in production, hindered by team silos, and grappling with complex, fragile systems.

Note: This book is currently in beta. As such, you may encounter content that is still being refined, sections that are being reorganized, or explanations that will be expanded in future updates. We welcome your feedback as we continue to improve and develop this resource.

This book is an endeavor to peel back the layers of CI/CD, moving beyond mere automation and frequent code merging. We aim to guide you through the intricacies of what CI/CD truly entails and how to implement it effectively using GitHub Actions as a practical example. While the theory of CI/CD promises streamlined operations and faster deployments, the reality involves navigating through a myriad of challenges that can stymie even the most determined teams.

What is CI/CD?

Continuous Integration (CI)

Integration is the act of constantly merging your changes with other developers', and vice-versa. It's the act of combining multiple changes, from multiple developers, into a single, cohesive whole, regularly. All developers work on a shared codebase. The product owner or another person (internally) should be able to use your app, or another team can demo their feature--it might not be finished but the application still works as intended.

Continuous Deployment (CD) and Continuous Development (CD)

Continuous Deployment (often confused with Continuous Delivery) is the practice where every change that passes the automated tests and other confidence-inducing procedures is automatically deployed into the production environment with little to no human intervention.

Continuous Delivery, on the other hand, ensures that the code is always in a deployable state, but it may not be deployed to production automatically. Instead, it might require manual approval. It provides the business with the opportunity but not the obligation to deploy at any point. Continuous delivery is not simply an automated pipeline for on-demand deployment. For example, code in long-lived feature branches necessitates retrieving specific versions or bug fixes that require complex version control, which can disrupt other work. Or, the build requires a special ceremony, such as complex testing, an implicit contract with another service that has to be deployed in a certain order, manually run scripts, manual checks, etc. This indicates the code base is not always deployable, thus not fully meeting continuous integration principles. This also includes necessary automated testing to ensure its capacity to be deployed continuously.

Deployments are technical events managed by engineering, releasing (making those features usable by customers) is both an engineering and a business task.

CI/CD

CI/CD aims to avoid "integration hell" by ensuring continuous integration and either continuous delivery or deployment. Work is constantly merged into the main/master branch after it has been verified via code review and the continuous integration pipeline. This involves practices like trunk-based development, where all developers work on a shared branch, promoting constant integration and minimizing merge conflicts.

Aside: Some companies deploy 100 times a day, but more deploys aren't inherently better—they simply indicate a robust, automated process. Continuous deployment automatically releases every quality-approved change, reducing the gap between versions. This means smaller changesets, easier bug identification, and faster rollbacks, all of which help minimize profit loss. Ultimately, frequent deploys reflect strong operational practices and many quality measures, not a superior app.

A misunderstanding of CI/CD is that it's just a build pipeline that continually builds the software. CI/CD requires both technical and cultural shifts, including:

• Smaller work units: Breaking down features into independently deployable and testable components. This allows the features to be continually deployed, or behind a feature flag, while other features are being worked on. If all features are large and are on their own feature branch, then this defeats the point of CI/CD as the feature has not yet been integrated, that is, it does not co-exist with the rest of the application. Other developers are unable to build around it, and feature flagging is not possible. Idea transmission is still possible, and it is a myth that developers do not communicate with each other if not practicing CI/CD.

• Modular codebase: Facilitating localized changes without impacting the entire application. This allows other developers to not be blocked while a parallel feature is in development.

• Focus on rapid feedback: Prioritizing quick delivery of changes and gathering customer insights. If there is no need for fast customer feedback or to test changes, then moving to CI/CD becomes less important.

These shifts require that the application itself is modular and easy to modify, therefore, it could require code changes, depending on your application.

Some cases, such as rewriting the app to use another framework, may require feature branching or interrupting others' work.

Here is what the software development process looks like when using CI/CD. Note that many of these processes are automated.

Why is CI/CD important?

There are many reasons why a company or a project may use CI/CD. Core Benefits:

• Faster Development and Deployment: CI/CD enables rapid deployment of small changes, accelerating development and deployment cycles, allowing businesses to be more agile and responsive to customer needs.

• Improved Code Quality: Continuous integration, automated testing, and code review practices built into CI/CD processes lead to higher-quality code and more reliable software.

• Increased Collaboration and Transparency: CI/CD encourages collaboration between developers, operations, and QA teams, fostering shared understanding and transparency throughout the development lifecycle.

• Decoupling of Integration, Deployment, and Release: CI/CD separates these stages, allowing for flexibility in releasing features and testing in production without impacting users.

• Enhanced Confidence in Changes: Automated testing and build pipelines provide developers with a higher level of confidence in their code, reducing the risk of introducing bugs.

• Improved Estimation Accuracy: By deploying frequently, teams gain a better understanding of the development process, leading to more accurate estimations.

• Streamlined Workflow: Automation eliminates manual processes, smoothing workflows, and allowing developers to focus on core development tasks.

• Support for Experimentation and Innovation: Feature flags enable controlled experimentation and incremental rollouts, allowing teams to test new features and gather feedback without risking the entire application.

Despite these benefits, several challenges can hinder successful CI/CD implementation:

• Zero-Downtime Deployments: Achieving seamless deployments while managing resources and data integrity requires strategies like blue-green deployments, canary releases, and feature flags.

• Database Schema Impacts: Even small code changes can disrupt database schemas, necessitating schema migration tools and a disciplined approach to database management.

• Central Point of Failure: CI/CD creates a central point of failure that demands constant vigilance. Maintaining a "green" pipeline requires rigorous testing, code review, and ongoing maintenance to ensure stability and compliance. Do not rubber stamp PRs.

• Culture Shift: CI/CD requires a shift in mindset, emphasizing collaboration, shared responsibility, and open communication across teams. This will exaggerate any communication issues, if they exist.

• Continuous Learning: Teams must invest in ongoing training, keeping their skills up-to-date with evolving CI/CD technologies and security best practices.

• Clear Objectives: A lack of clarity regarding CI/CD goals can lead to resistance and misaligned expectations. It's crucial to define objectives, communicate the value proposition, and secure stakeholder buy-in.

CI/CD is not a magic bullet. It demands discipline, commitment to quality, and a proactive approach to addressing technical and organizational challenges. However, when implemented effectively, it can significantly accelerate development, enhance software quality, and empower teams to deliver value more efficiently.

Traditional software development

Traditional software development is a methodology that is difficult to define because there's multiple definitions of what traditional means. This usually means before continuous integration and development was widely popularized, for example prior to 2010.

Traditional Development:

• Teams often work in silos with limited visibility into each other's work. This does not mean that team members do not communicate with each other, rather, the act of integration is delayed.

• Slow feedback loops and long development cycles are common.

• Manual integration and deployment processes are complex and resource-intensive.

• Late-stage testing limits opportunities for early customer feedback.

CI/CD Development:

• Promotes continuous collaboration and transparency through practices like trunk-based development.

• Enables rapid feedback loops and iterative development with frequent integrations and deployments.

• Automates builds, tests, and deployments, freeing developers to focus on core tasks.

• Allows controlled feature rollouts and early customer feedback through feature flags.

The build server or build pipeline

A build server is a dedicated computer or virtual machine that automates tasks such as building, testing, linting, and conducting security scans, preparing code for deployment or integration. It acts as a quality gatekeeper, running CI/CD workflows before code is deployed or merged into the main branch. The build server doesn't inherently perform tasks but executes the instructions specified in the workflow file by developers. Anything can be run on a build server, since it's just a virtual machine.

Build servers are used instead of developer workstations because:

• Security: These servers handle sensitive resources like company source code and secrets. It is crucial to secure them to prevent unauthorized access and protect against lateral attacks. Simply storing them on a developer's machine means that other software could use the secrets, the secrets are transmitted over other mediums, etc.

• Consistency and Isolation: Each server, agent, or VM should operate independently to minimize the impact of potential compromises. The agent only runs for a fixed amount of time, then is erased. Developer machines are long-lived, and could have lots of software unnecessary for building the application.

Automation

Automation is essential for CI/CD, streamlining tasks like builds, deployments, and testing. This saves time, improves efficiency, and ensures consistency and reliability, crucial for frequent deployments. However, over-automation can be detrimental, especially for tasks requiring human judgment or adaptability.

The key is to find the right balance, automating repetitive tasks while retaining human oversight for critical decision-making and complex scenarios. Robust error handling and clear guidelines for human intervention are crucial for successful automation.

[researchgate.net/profile/Kevin-Hoff-3/publication/272887576_Trust_in_Automation_Integrating_Empirical_Evidence_on_Factors_That_Influence_Trust/links/57952ba008aec89db7a8cf4f/Trust-in-Automation-Integrating-Empirical-Evidence-on-Factors-That-Influence-Trust.pdf]{.underline}

Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics, 37(11), 1905--1922 | 10.1080/00140139408964957

A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 30(3), 286--297 | 10.1109/3468.844354

Testing, code review, and quality assurance

Testing and quality assurance are crucial for CI/CD, ensuring software quality and confidence in deployments. While both automated and manual testing play vital roles, they address different aspects:

• Automated Testing: This process verifies functionality and performance through predefined tests, similar to controlled experiments, providing rapid feedback on code changes. Imagine a chemistry teacher at the front of a classroom, mixing two chemicals and instructing students to watch closely. This scenario serves as an example of a demonstration because the outcome is known beforehand, akin to how these tests predictably assess the impacts of changes in the code.

• Manual Testing: Leverages human judgment for evaluating usability, aesthetics, and user experience, crucial for aspects difficult to automate.Humans should not be doing the checking aspect.Rather, automated testing should be responsible for that.

• CI/CD emphasizes automation but doesn't eliminate the need for manual testing. Instead, it allows testers to focus on higher-level tasks requiring human expertise. Maintaining a balance between automated and manual testing is key for efficient, high-quality software development.

• Skipping quality assurance in CI/CD can be tempting due to the fast-paced nature, but it's essential for ensuring customer satisfaction and protecting the business's reputation.It is additionally very tempting because the lack of automation will not show up for quite some time.

Aside: fire QA, right?! Well, no. QA shifts left, and instead prioritizes testing PRs (which have a smaller scope and smaller changeset.) Since checking (testing an outcome that is known) is done mostly via unit tests, QA can use their human-ness to evaluate the product for quality, usability, functionality, and exploration testing. When a feature is developed under a feature flag, QA can test it in the pre-production environment (feature flag enabled for them), allowing developers to get early feedback.

Rapid Feedback Loops

The essence of CI/CD lies in maintaining business agility through a fast feedback loop. This allows companies, especially startups and small businesses, to rapidly experiment, identify what works, and make quick improvements.

Rapid feedback loops are a multi-pronged approach:

• Streamlined Local Testing: Developers need easily accessible local testing environments mirroring production. Tools like hot reloading and ephemeral test environments with simple provisioning are crucial.

• Efficient Build Pipeline: Aim for a build time under 15-20 minutes with automated processes, notifications for failures, and minimal manual intervention.This period of time is arbitrary. However, if the build time is too long, then there's a possibility of frustrating developers as well as not being able to quickly react to feedback from your customers.You will also make it more difficult to quickly push changes should there be a production outage.

• Timely Code Reviews: Prioritize prompt and thorough PR reviews (ideally within a day) with constructive feedback and a focus on code readability.

• Regular Deployments: Embrace semi-regular deployments to accelerate feedback loops and customer value delivery (refer to DORA metrics).

• Comprehensive Monitoring & Alerting: Implement robust monitoring in all environments to detect issues early. Define a severity matrix for appropriate stakeholder notifications, escalating critical incidents promptly.

Infrastructure as Code and modularity

To achieve continuous integration and efficient deployments, it's essential to structure applications so that small changes are manageable. This involves both the application itself and its underlying infrastructure. If making small changes is cumbersome, integration becomes challenging, as larger updates can span multiple components, increasing both the testing burden and the associated risks.

• Independent Modules: Structure applications with clear boundaries between components. This facilitates isolated changes and reduces testing complexity. This isn't the fact that you must adopt microservices, rather it's just structuring your code to be a modular approach. Modularity leads to smaller, more manageable changes, simplifying testing and increasing development speed.

• Version-Controlled Infrastructure: Treat infrastructure configurations like code, storing them in version control systems for tracking, reverting, and collaboration.Your application. This could be terraform templates or ARM templates.

• Eliminate configuration inconsistencies between development, testing, and production, preventing "snowflake servers" and ensuring reliable deployments.

Feature Flags

Feature flags are for experimentation and release. They separate the act of deploying (moving the code to production, managed by engineering) and the act of making the changes usable by customers (commonly associated with a marketing event from the business's side.) They are remote-controlled conditional statements that allow the selective activation or deactivation of application functionalities across different environments (development, integration, pre-production, production) without needing a redeployment. These flags can be toggled within seconds or minutes and can be set based on criteria like geographic location, IP address, or user type, facilitating targeted and gradual feature rollouts.

What exactly constitutes a feature or needs to be released via a feature flag is up to the product managers and the business. Usually not everything is behind a feature flag, for example, features that cannot be code compiled, those that are incomplete to the extent that they have security issues that could harm the product, logging statements, refactors, package upgrades, security fixes, bug fixes, or small changes like typo fixes.

Typically, developers can enable these feature flags by themselves. Here's an example of an application in development, and it shows a special development overlay that allows developers to toggle feature flags.

[Implementing feature flags in React with Unleash - Case Study (claimcompass.eu)]{.underline}

Feature flags need not be complicated or require third-party software. You can get started with a simple JSON file with a list of key/value pairs that is outside of the deployment system, but still accessible by your app. This does not require any subscription to a feature flag service. They can also be embedded in your application, for example, in a config file. This approach limits flexibility, however, as a redeployment is needed to change the config file.

This approach is beneficial for trunk-based development, where changes are incremental. Developers can merge new features behind feature flags, allowing others to activate these flags for testing selectively.

Feature flags also enable controlled risk-taking. For example, a promising feature might be released to a small user segment (e.g., 1%) to evaluate performance and gather feedback, minimizing risks of broader release.

Branches versus Feature Flags:

Branches provide isolated workspaces for developers, supporting multiple application versions or development paths. However, unlike branches that delay integration, feature flags allow for integration while controlling feature activation.

Limitations:

Feature flags should not be used to restrict feature access (for example, paid features), as they are often visible and modifiable on the client-side. They are better suited for testing, phased rollouts, and controlled changes.

Maintenance:

Proper feature flag management is crucial. Unused flags should be removed to avoid clutter and potential confusion. Limiting the number of active feature flags helps reduce code complexity and ease debugging.

Summary table,

Version Control System (VCS)

Version control systems are crucial for continuous integration and development because they track changes, simplifying the integration process. For instance, if you have two versions of a document, merging them into one requires a detailed comparison of each word. This task involves identifying and understanding changes. Version control automates this process, significantly reducing the likelihood of errors associated with manually tracking changes. This automation ensures smooth and accurate integration of code changes, forming a cohesive whole.

VCSes show that work has been integrated because it is considered a central source of truth. Multiple copies of the application with different versions mean that there isn't a single source of truth, therefore, we can't know if our changes have been integrated.

VCSs enhance auditability, allowing developers to easily trace back to see when and why code was altered. This is particularly important from a security perspective to ensure that only authorized changes are made. For example, if unauthorized changes occur, they can be quickly identified and reverted.

Culture and communication, collaboration

While CI/CD tools automate integration and deployment, successful implementation requires more than just technology. It demands a fundamental shift in organizational culture and project management.

CI/CD thrives on:

• Collaboration and Communication: Teams must work closely, sharing information and coordinating efforts to ensure smooth integration and deployment.

• Rapid Iteration: Frequent code merges, small feature updates, and continuous feedback loops are essential for maximizing the benefits of CI/CD.

• Strategic Project Management: Breaking down features into manageable, independently testable units facilitates continuous integration and deployment without disrupting the entire application.

Ignoring the human element of CI/CD can lead to challenges:

• Batched Changes and Integration Conflicts: Infrequent code merges increase the risk of complex integration issues.

• Delayed Feedback: Waiting to test in production hinders rapid iteration and learning.

• Siloed Information and Debugging Difficulties: Poor communication can lead to significant debugging challenges.

CI/CD is not a one-time setup. It requires ongoing maintenance, pipeline updates, and continuous learning to adapt to evolving practices. Effective testing, code reviews, and organizational support for these processes are vital for maintaining a smooth development cycle.

Continuous Deployment/Continuous Delivery

Infrastructure as Code (IaC) represents a transformative approach in managing and provisioning computing resources, utilizing machine-readable definition files rather than traditional physical hardware setups. This automation-focused methodology enhances the setup, configuration, and management of infrastructure, promoting rapid deployments, efficient resource utilization, and consistent, reliable environments. IaC is mainly declarative, targeting the desired end state of the infrastructure while the underlying tooling manages the execution. This is crucial in Continuous Deployment (CD) pipelines where IaC templates are automatically deployed in the cloud, ensuring each deployment is consistent, reproducible, and easily reversible. This aligns with principles like idempotency, immutability, and composability---key for maintaining interoperable and stable components.

The benefits of adopting IaC are extensive, including consistent infrastructure deployments across environments, enhanced reproducibility, and robust version control which acts as a single source of truth. Such structured deployments reduce configuration drifts between different environments such as QA/dev and production, speeding up the feedback loop for developers and boosting security measures. Tools such as Terraform offer cloud-agnostic deployment options, whereas AWS CloudFormation, Azure Resource Manager, and Google Cloud Deployment Manager cater to specific cloud environments. Additionally, open-source tools like Ansible and traditional configuration management tools like Chef and Puppet provide further automation capabilities, ensuring thorough enforcement of system states.

Historically, server management was a manual process involving system administrators physically logging into servers to apply changes, a method prone to errors and inconsistencies, especially in complex server environments. This labor-intensive process made replicating servers difficult, often requiring extensive documentation and manual reconfiguration. Before the adoption of IaC, administrators relied on shell scripts to manage and synchronize server configurations, though these scripts were limited in handling complex scenarios effectively. The rise of configuration management tools in the mid-to-late 2000s, such as CFEngine, Puppet, and Chef, began to address the issue of "snowflake servers"---highly customized servers difficult to replicate from scratch. Despite the advancements, many continued using shell scripts and command-line tools for their simplicity and familiarity. Today, IaC practices, exemplified by Terraform and other cloud-specific tools, have largely superseded these older methods, providing scalable, reliable, and repeatable server environment setups.

Here's a snippet of a simple Terraform configuration that demonstrates how to create an AWS infrastructure:

provider "aws" {
  region = "us-west-1"
}

resource "aws_vpc" "sample_vpc" {
  cidr_block = "10.0.0.0/16"
  ... // Additional configurations
}

resource "aws_subnet" "sample_subnet" {
  vpc_id     = aws_vpc.sample_vpc.id
  cidr_block = "10.0.1.0/24"
  ... // Additional configurations
}

resource "aws_instance" "sample_ec2" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
  subnet_id     = aws_subnet.sample_subnet.id
  ... // Additional configurations
}

Providers and hosting

Providers fall into two overlapping categories: CI/CD tooling and application hosting infrastructure. You can mix and match—for example, use GitHub Actions for CI/CD while hosting your application on AWS, Azure, or Google Cloud. Sticking with a provider you already have a contract with can streamline integration with your security policies.

CI/CD Tools:

• GitHub Actions: Built into GitHub for automated build, test, and deployment workflows.
• GitLab CI/CD: An integrated solution with built-in CI/CD and version control.
• Jenkins: A flexible, open-source automation server with a vast plugin ecosystem.
• CircleCI: A cloud-based service known for ease of integration, Docker support, and parallel builds.
• Azure DevOps: A comprehensive suite covering planning, coding, building, and deploying.
• Bamboo (Atlassian): Integrates with Jira and Bitbucket, ideal for enterprises using Atlassian tools.
• Travis CI: A hosted service that integrates well with GitHub and Bitbucket.

Infrastructure Providers:

• AWS: Offers comprehensive cloud services with CI/CD tools like CodePipeline and CodeBuild.
• Azure: Provides robust hosting alongside Azure Pipelines and other DevOps services.
• Google Cloud Platform: Features Cloud Build and strong support for containerized workflows.
• IBM Cloud: Delivers end-to-end DevOps toolchains integrated with popular tools.
• DigitalOcean: A straightforward platform that supports Kubernetes and common CI/CD integrations.

Terminology

This book is somewhat focused on GitHub Actions, but tries to provide a provider-agnostic view. Some of the terms might be a bit different depending on your CI/CD provider. Here is a table that helps clarify.

Further readings

[Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation: Humble, Jez, Farley, David: 9780321601919: Books - Amazon.ca]{.underline}

[Continuous Integration: Improving Software Quality and Reducing Risk: Duvall, Paul, Matyas, Steve, Glover, Andrew: 9780321336385: Books - Amazon.ca]{.underline}

[The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win: Kim, Gene, Behr, Kevin, Spafford, George: 9781942788294: Books - Amazon.ca]{.underline}

[The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations: Kim, Gene, Debois, Patrick, Willis, John, Humble, Jez, Allspaw, John: 9781942788003: Books - Amazon.ca]{.underline}

Thanks for reviewing my book!

What I am looking for:

• Sections that should be removed (have no relevance to the book at all), added, or are debatable (provide a case study or references).
• What sections you'd like me to write more about, what sections are neutral, and what sections are boring.
• High-level overview of any organization changes (e.g., rearranging the sections in the table of contents).
• Technical inaccuracies (when possible).
• Changing the table of contents to better align with the audience.
• Whether you would recommend this book to your friends (I promise I won’t hold you to it).
• Whether the current table of contents may be sufficient to meet the page count goal, or I may need to write more about other sections.

What I am not looking for as much:

• Spelling, grammar, or formatting, unless formatting makes the text unreadable.
• Nit-picks.

Target Audience: Software developers, system administrators, and DevOps professionals with 1-3 years of experience, seeking to implement efficient CI/CD practices in startups and small businesses.

Focus: This practical guide provides a streamlined approach to setting up and managing CI/CD pipelines with limited resources, emphasizing business needs and rapid deployment.

Key Features:

• Advanced Beginner/Intermediate Level: Goes beyond introductory concepts, catering to developers with existing programming, testing, and Git experience.
• Cost Management Strategies: Practical tips for optimizing runner usage, leveraging free tiers, and avoiding unexpected billing.
• GitHub Actions Focus: Detailed walkthroughs and pitfalls of GitHub Actions, with a brief overview of other CI/CD providers.
• Efficient Pipeline Setup: Streamlined workflows, avoiding common pitfalls and unnecessary debugging, with a focus on business value.
• Trunk-Based Development: Emphasis on frequent deployments and rapid integration, with concise mentions of alternative branching strategies.
• Security Best Practices: Basic security scanning tools and techniques, secrets management, and prioritization of security alerts (e.g., Dependabot).
• Effective Testing Strategies: Writing impactful tests, managing manual testing, and aligning test strategies with business goals.
• Practical Deployment and Monitoring: Deploying updates quickly, handling rollbacks, and understanding the importance of continuous monitoring.
• Automation vs. Manual Processes: Identifying what to automate and what to keep manual in a dynamic startup environment.
• Real-World Context: Touches on Agile methodologies, regulatory considerations (e.g., FDA), and the HP case study for embedded systems.
• Emphasis on Practicality: Real-world scenarios, such as receiving phone alerts for production issues, and adapting CI/CD to dynamic environments.

This book provides the essential knowledge and practical skills needed to successfully implement and manage CI/CD, empowering developers to streamline their workflow, accelerate deployments, and improve software quality.

Book will have about 350 pages when complete.

Getting started with the workflow

In order to have a good understanding of how CI/CD works, it's important to have a good idea of how everything fits in together. Here's the overall process, at a very high level view, from working on a feature to getting it out into production.

Developers create short-lived branches for their work:

• This isolates their changes from the main codebase ("trunk") and allows collaboration with other developers.Short lived in this context refers to just the minimal amount of time needed to work on that particular task and no more.This means that the tasks have to be broken down sufficiently as well as broken down in such a way where the tasks are easy to complete and don't interfere necessarily with other commits. They're also testable as well, capable of showing that they can have the capacity to be integrated. You can look at the appendix For more information on how to break down tasks into such a way that makes them.To show that they've been integrated successfully.

• Branches can be created locally or through GitHub's UI.

Pull requests (PRs) are used to merge code into the trunk:

• Developers create PRs when they're ready to integrate their changes.They create the PR from their branch.

• This triggers a pipeline that builds, tests, and runs the code to ensure quality.The pipeline and its trigger must be set up by developers beforehand. By default it does not do this.

• PRs require review and approval before being merged.Normally to approvers are required, but it depends on your team.At least one approver should approve.

Merging PRs updates the trunk:

• This makes the changes available to all developers.

• Developers need to pull the latest changes into their branches to stay up-to-date.This is not a requirement as when you create the PR it will automatically merge their branch into the trunk when building.However, it's recommended that developers pull their latest changes to keep up to date because the merge might be different than what they tested locally, potentially introducing the possibility of bugs.

Branch Management:

• Short-lived branches are typically deleted after merging.

• Long-lived branches are useful for experiments and large refactorings, such as migrating frameworks (e.g., Spring to Hibernate). However, these situations are infrequent. Most development (90-95%) utilizes short-lived branches to ensure continuous integration, cohesion, and comprehensive testing.

• Descriptive branch names, including developer IDs (e.g., "username/feature-name"), improve organization and maintenance. This convention aids in automatic categorization within popular CI/CD platforms like Azure DevOps, grouping branches into logical directories.

Build Server:

• The build server clones the developer's branch associated with the PR and builds the code.In this case the build server is provided to you by GitHub Actions, but you can also use a self hosted runner yourself.

• This ensures that the changes are compatible with the existing codebase.It's important to write good tests as well as making sure that the bell script is up to date to make sure that the pipeline sufficiently instills confidence in your changes. The build pipeline is only as useful as the effort and criteria that you put into it. It is not magical.

What is a repository?

• A repository is a way to store a collection of files that are used in the build process, and should include all files that allow the application to be built, including configuration files, database migrations, tests, etc. Developers work off of a branch, which is a snapshot of the repository. Each repository is usually a deployable unit, and normally should not include other projects unless they are part of the same application or must be deployed together. It is managed with a VCS (e.g., Git.)

• However, it should not include environment-specific files, secrets (such as passwords), API keys, or files specific to a single developer's environment because these are typically injected at runtime, and should not be part of the application, as they could be erroneously leaked to production.Normally injected at runtime. Do not inject it in your continuous integration and development server.

• Also, if a single developer's settings are in the repository, it's not much use for the other developers and could cause confusion.

Typical development scenario using VCS

A developer works on code using their IDE, on their development branch, from their repository as shown in this screenshot. This is the code repository as discussed earlier. VS Code is a free IDE that is open source.

[TypeScript Programming with Visual Studio Code]{.underline}

While git can be easily used on the command line, sometimes developers prefer a GUI-based approach. This can be especially helpful for those who are new to git. Some applications can help with this, such as "GitHub Desktop".

While the developer is coding, they normally run unit tests or integration tests in their local environment, which is part of having a fast feedback loop. These tests exist as part of the repository. This provides them confidence for their changes. If they find a broken or failing test, then they would fix it on their development workstation before proceeding, because otherwise the pipeline would fail and they wouldn't be able to merge their code. Developers are responsible for writing and maintaining tests.

[Rust Test Explorer - Visual Studio Marketplace]{.underline}

For instance, consider a repository opened in GitHub Desktop. It's a tool to visualize and interact with a Git repository, showing individual changes and their details. While this isn't a tutorial on Git, it's worth noting that any VCS with the capability to track and manage changes suffices; it doesn't have to be Git. The choice depends on your team's preferences and needs. Here is a screenshot of GitHub Desktop:

[https://github.com/desktop/desktop]{.underline} Some other GitHub Desktop-like tools are SourceTree, GitKraken, Tower, and TortoiseGit.

SourceTree – Free for Windows and Mac.

• Offers a visual representation of branches and commits.
• Integrated with Bitbucket and GitHub.
• Can be slow and occasionally buggy.
• Somewhat steeper learning curve for beginners.

GitKraken – Intuitive UI, great for beginners.

• Cross-platform (Windows, Mac, Linux).
• Supports GitFlow.
• Free version has limitations; Pro version is paid.
• Some users report performance issues.

Tower – Clean UI and efficient performance.

• Offers advanced features like single-line staging.
• Good integration with multiple services.
• No Linux version.
• More expensive compared to other tools.

TortoiseGit – Integrates directly into Windows Explorer.

• Free and open source.
• Mature and well-maintained.
• Windows only.
• UI might not be as modern or intuitive as other tools.
• Requires separate Git installation.

After developers are done, they then create a PR. This shows the developer's changes and allows other developers to comment on them (i.e., code review.)

What is a pull request?

A pull request (or a change request) is a request from a developer to merge changes from their branch into the repository. For example, when a developer is ready to integrate their changes, they would create a pull request. The pull request allows others to comment on the developer's changes and also triggers the build pipeline to validate their changes. Since a developer needs to create a pull request before merging, this means that the confidence-instilling checks can run.

After the pull request is merged, then the changes become part of the "main" branch in the repository. This means that they can be deployed, or other developers can integrate on top of their work.

Here are some key characteristics of a good pull request:

• Clear Title: The title should be concise and describe the changes at a high level. Someone should be able to get an idea of what the PR is about just by reading the title.

• Descriptive Description: A PR description should provide context. It should answer:

• - What changes are being made?

• - Why are these changes necessary?

• - How have these changes been tested?

• - Are there any related issues or PRs?

• Small and Focused: Ideally, a PR should be small and address a single issue or feature. This makes it easier to review and understand. Large PRs can be daunting for reviewers.

• Includes Tests: If the project has a testing framework, the PR should include tests that cover the new functionality or bug fixes. This ensures that the changes work as expected and prevents regressions.

• Follows Code Style and Conventions: The PR should adhere to the project's coding standards and conventions to maintain consistency.

• Updated Documentation: If the changes introduce new features or modify existing ones, the PR should also update related documentation.

• Comments on Complex or Ambiguous Code: If the changes involve complex logic or hacks to address specific edge cases, they should be accompanied by comments explaining the rationale.

• Includes Relevant Assets: If the PR has UI changes, including screenshots, gifs, or videos can be very helpful for reviewers.

• Has Been Self-reviewed: Before submitting, the author should review their own PR. This can catch many small issues like typos, console logs, or forgotten debug statements.

• Passes Continuous Integration: If the project uses CI tools, the PR should pass all checks (like building without errors, passing tests, lint checks, etc.)

• Addresses Feedback: After receiving feedback, the PR author should make the necessary changes and might need to clarify if something isn't clear. A good PR evolves through collaboration.

• Links to Issue: If the PR addresses an open issue, it should link to or mention that issue. This provides context and allows for tracking the resolution of bugs or features.

[How To Create a Pull Request With GitHub Desktop (vonage.com)]{.underline}

Testing and automated testing in the pipeline

Software testing is crucial for ensuring that applications meet quality standards, function correctly, and deliver a positive user experience. It's a vital part of CI/CD because it helps developers catch bugs early and build confidence in their code changes. When the build pipeline runs, it runs your automated tests. If the automated tests fail, then therefore the build pipeline will fail as well. This is important, because this tells the developer that their changes cannot be merged (i.e., integrated and available to other developers) because something is wrong.

There are two main categories of testing:

• Automated Testing ("Checking"): These tests are predefined (automated tests), can be run by computers, and are coded by developers. They are essential for verifying functionality and performance but very tedious, time-consuming, expensive and boring for a human to do manually.

• Manual Testing: This involves human judgment and focuses on aspects like usability, aesthetics, and user experience that are difficult to automate.

Checking is like a demonstration. Imagine a chemistry teacher in front of an eager class. The teacher says, "Watch this!" and mixes two chemicals together. Then, everything changes color and makes a pop. The teacher already knew what was going to happen, and in this case demonstrated that fact in front of a class: the chemicals were going to change color, and make a pop. Or, say someone enters in 1+1 in the calculator. There is an expectation that it will always be "2".

Now, contrast this with manual testing: tacit knowledge. You have to write a set of rules to determine if a website is designed well. It's not very easy to write a set of rules, or instructions for someone to determine that. But, it is easy to figure out using our human brains: we try to use the website, and if we are having issues, then we know that it's not usable. But we can't create a document that describes every possible facet in great detail for every possible situation of what usable means. It's highly subjective and complex. It's important to have both types of testing.

There are many types of automated tests, including:

• Unit Testing: Verifies individual components of code.

• Integration Testing: Checks how different units of code work together.

• Functional Testing: Tests the application against business requirements.

• System Testing: Examines the fully integrated software product.

• End-to-End Testing: Simulates real-user scenarios.

• Acceptance Testing: Determines if the software is ready for release.

• Performance Testing: Assesses speed, response time, and stability.

• Stress Testing: Evaluates how the system performs under extreme conditions.

• Security Testing: Identifies vulnerabilities.

• Usability Testing: Evaluates user-friendliness, which may or may not include a human reviewer.

• Regression Testing: Ensures new changes don't break existing functionality.

• Smoke Testing: Identifies major failures early on.

• Exploratory Testing: Involves less structured exploration of the software.

Getting started with GitHub Actions

Throughout this guide, we will explore the key features of GitHub Actions and how to effectively structure workflow files in YAML to maximize the benefits of CI/CD. We'll start by creating a somewhat simple weather application, but make it more complex over time. This is designed to simulate a real world application.

GitHub is a company that has a product called "Actions" (sometimes referred to as "GitHub Actions") that is a set of build servers and software that runs GitHub Actions workflows. These YAML workflows are created by the developer and normally build, test, and lint the code using the GitHub Actions YAML syntax and run on the GitHub Actions build servers.

GitHub is a company, thus, it is not possible to install GitHub--it is not an application. Git is the version control system that can be installed.

If you need more information on the specific intricacies of GitHub Actions, please see the GitHub Actions documentation. [GitHub Actions documentation - GitHub Docs]{.underline}

Let's build a pipeline that can do the following:

• Checkout the code (i.e., clone it onto the runner.)

• Build the code.

• Run automated tests, and linting.

• Publish artifacts to an artifact server, in this case, to GitHub, along with a versioning strategy that will help identify which artifacts you are publishing.

• Deployed the website to Azure.

Workflow Structure

Here's an overview of how GitHub workflows are structured:

• 1. Events: Workflows begin with events, such as pushes or pull requests, which trigger the workflow.

• 2. Jobs: Workflows may contain multiple jobs, but we will focus on a single job for simplicity. Each job specifies an environment to run in, indicated by a string that corresponds to an operating system and a pre-configured image. This image includes pre-installed software, allowing us to get started quickly and reduce setup times and costs.

• 3. Steps: Each job is composed of multiple steps. These steps can use either the uses or run command:

• - Uses: This command utilizes actions provided by GitHub Actions, sourced from the GitHub Marketplace. These actions are pre-configured scripts that handle tasks like software installation, version management, or building.

• - Run: This command executes shell commands specific to the operating system defined in the job's environment, using bash scripting for Linux, for example.

• 4. Artifacts: Typically, workflows end with steps for uploading artifacts, though the initial steps may also involve downloading or preparing artifacts.

Below is an overview of a typical workflow structure:

Workflow
│
├── Events (e.g., push, pull_request)
│
├── Jobs
│   ├── Runs-on (Runner)
│   ├── Needs (Dependencies on other jobs)
│   ├── Steps
│   │   ├── Uses (Actions)
│   │   │   ├── Inputs
│   │   │   ├── Outputs
│   │   │   └── Environment (e.g., secrets, env variables)
│   │   └── Run (Shell commands)
│   ├── Environment Variables
│   ├── Secrets
│   ├── Services (Service Containers)
│   └── Artifacts
│       ├── Upload Artifact
│       └── Download Artifact
│
└── Workflow Commands (e.g., set-output, set-env)

If you want to get started right away, GitHub Actions has several templates for many different project types. Use a template to get started quickly.

Setting up error notifications

When your workflow fails, it means that continuous integration is no longer possible. Implement error notifications to alert your team when the build pipeline fails—especially for production workflows. Consider these notification methods:

• Email Notifications: Configure GitHub Actions to send emails upon failure.
• Messaging Platform Integrations: Integrate with platforms like Microsoft Teams, Slack, or Discord to receive instant alerts (including texts or phone calls).

Ensure your GitHub email settings are correctly configured to receive these notifications. GitHub Actions is a CI/CD platform that automates software development tasks within GitHub repositories. It uses "workflow files," which are YAML-based instructions that define the steps of a CI/CD pipeline, similar to a project manager for your build scripts.

These workflows are triggered by specific events in your repository, like pushing code or creating a pull request. When triggered, they run on virtual build servers provided by GitHub, executing tasks such as building, testing, and deploying your application. These servers are ephemeral -- they're created for each workflow run and deleted afterward, ensuring a clean and consistent environment.

Workflows are organized into "jobs," each containing multiple "steps." Each step represents a discrete action, like running a script or using a pre-built action from the GitHub Marketplace.

Benefits of this structured approach:

• Clarity and Organization: Named steps improve readability and make it easier to track progress, debug issues, and set up notifications.

• Security and Isolation: Steps run in isolated environments, protecting sensitive information like secrets and environment variables.

• Efficiency and Automation: GitHub Actions provides features for parallelization, triggering, resource management, and secret management, simplifying complex tasks.

• Standardization and Collaboration: The workflow syntax promotes consistency across projects and teams, facilitating collaboration and knowledge sharing.

[5 Things to Know About Pipe Scaffolding (supremepipe.com)]{.underline}

In this example, we demonstrate how you can execute commands on your local computer to simulate what a build server does. You can effectively use your own laptop as a server, albeit with caveats mentioned earlier. As an exercise, consider installing the GitHub Actions agent on your computer. Then, set up a self-hosted runner and execute the build script on it. This process will allow you to recreate or emulate the actions performed by a build server, right from your local environment. See the appendix for more info.

Aside

The script echo hello world is a Bash script. Note that while Bash is commonly used, some scripts might be written for sh, which has slight syntax differences. For Windows runners, remember that these execute PowerShell scripts—not Bash scripts. This guide does not cover PowerShell in detail, but if you are new to Bash, consider reading a beginner’s guide. Given Bash’s long-standing usage, it’s likely to remain relevant for some time.

The feedback loop for workflow changes can be slow—you typically need to edit, commit, and run the workflow on GitHub Actions to see the results. To streamline this process, consider these strategies:

1. Simplify Workflow Steps:
   Ensure that workflow steps are simple enough to run locally. This improves speed and manageability. Use provider-agnostic scripts (e.g., PowerShell or Bash).

2. Use Docker Containers:
   Create and use a Docker container that closely mirrors the GitHub Actions environment. This lets you test workflows locally in a similar setting.

3. Utilize the act Library:
   The act library lets you run GitHub Actions locally. While it may not perfectly replicate the GitHub Actions environment, it works well for simpler scripts. See the appendix for more details.

Aside end

Workflow files must be stored in the .github/workflows directory of your repository. This YAML file dictates the sequence of operations executed by GitHub Actions during the CI/CD process.

In order to run a workflow, you need a GitHub account and potentially a GitHub Enterprise organization. To create a new GitHub Enterprise repository, you first need to set up an account on GitHub and potentially get access to GitHub Enterprise, depending on your organization's setup. Here's how you can do it step-by-step:

1. Sign Up for GitHub

• Go to [GitHub](https://github.com/).

• Click on the "Sign up" button at the top right corner of the page.

• Fill in your details, including a username, email address, and password.

• Verify your account by following the email verification link sent to your email.

2. Join or Set Up GitHub Enterprise

• If your organization already has GitHub Enterprise: You will need an invitation to join from your organization's GitHub Enterprise admin. Once invited, you can log in using the credentials or SSO (Single Sign-On) method prescribed by your organization.

• If you are setting up a new GitHub Enterprise: You can start a trial or purchase it by visiting the [GitHub Enterprise page](https://github.com/enterprise). Setting up GitHub Enterprise usually requires more extensive IT involvement to handle the installation on cloud or on-premises infrastructure.

3. Create a New Repository

Once you have access to GitHub (and GitHub Enterprise if using):

• Click on your profile photo in the upper right corner, then click Your repositories.

• Click the green New button, or if you are on your organization's GitHub Enterprise account, you may need to select the organization context first.

• Enter a repository name, description (optional), and decide if the repository will be public or private.

• Configure other settings like adding a README file, .gitignore, or a license according to your project needs.

• Click Create repository.

4. Clone the Repository

• After creating your repository, clone it to your local machine to start working on the project. You can do this by opening your command line or terminal and running:

git clone https://github.com/username/repository-name.git

Replace username and repository-name with your GitHub username and the new repository's name.

5. Start Pushing Code

• After cloning the repository, you can start pushing your code to the GitHub repository by using:

git add .

git commit -m \"Initial commit\"

git push -u origin main

An exercise, try putting in the workflow described earlier in the doc GitHub workflow and see if you can figure out how to run it. It should just display Hello World and it doesn't require any code to build.

Workflow Triggers and Patterns

Workflow triggers are ways to automatically trigger your pipeline. When the workflow is triggered, it receives a set of inputs, for example the branch that it was triggered on as well as the date and time.

We'll be using a trigger to automatically trigger when we make a pull request. This means that the pull request will be blocked until the pipeline passes.

These pipelines can also be brought on different branches and triggers, for example, any pushes to the main branch. For example, if you're practicing continuous deployment, you may want to automatically deploy changes that are pushed to the main branch. Therefore, you can add a trigger that will automatically run the workflow if there's a commit push to the main branch.

on:

pull_request:

push:

Just keep listing the items if you want to listen to more events. Note that the "push" event also accepts many options to narrow it down. It also ends with a colon because you can narrow it down with more filters.

Triggers are not isolated; they're evaluated as a set of rules within the workflow file under the on: key, where multiple events like pull_request and push can be listed. This setup allows the workflow to execute under various conditions but can be refined to ensure efficiency and relevance. The workflow runs when at least one of those events are triggered.

Order doesn't matter. I could write it like this, it still works:

on:

push:

pull_request:

Aside start

It's important to configure workflow triggers to respond only to relevant events, helping to prevent unnecessary runs and reduce costs. For example, a trigger set for "pull_request" events can automate tasks like code integration and deployment specifically when changes are proposed to a main branch. To avoid redundant executions in environments with active development, you should define triggers carefully by specifying branches, tags, or paths.There's some more information in the appendix about common files that are typically ignored when changed to prevent excessive pipeline runs.

Aside end

Setting Up Your First Workflow

To create a basic "Hello world!" workflow in GitHub Actions, start by creating a new file named main.yml in the .github/workflows directory in your previously created branch and add the following content:

name: Hello World Workflow

on:

workflow_dispatch:

jobs:

greet:

runs-on: ubuntu-latest

steps:

- name: Say Hello

run: echo \"Hello world!\"

This example introduces the workflow_dispatch trigger, which allows you to manually start the workflow. This feature is particularly useful for debugging purposes. The workflow is set to execute on ubuntu-latest, a Linux-based runner that utilizes the Bash shell---a standard configuration for many GitHub Actions workflows.

Here are some tips for the workflow:

• Steps in a workflow are used to organize similar scripts or commands, grouping them together for logical execution. Each step in the workflow is executed sequentially, one after the other. To enhance the auditability of the workflow and simplify the debugging process, it is beneficial to keep each step as concise as possible. This approach not only clarifies the structure of the workflow but also makes it easier to identify and resolve issues within specific steps.

• Tips for the YAML syntax:

  • Indent with two spaces.

  • Use : to indicate options for a key.

  • Quote values to ensure they are interpreted as strings.

  • Validation: Use a YAML linter or language server to avoid syntax errors.

  • For more information see the sample web page called Learn YAML in X minutes.

Now commit this file and then push those changes to your branch. You should see the following screenshot.

You should see "Hello World Workflow" in the sidebar on GitHub. Run it and check the output.

After you've run it, then you should see the output.

Workflow Name

Triggers

Jobs

For now, everything goes in a single job called all.

Popular options:

• ubuntu-latest (Linux): Supports bash and cross-platform scripts (e.g., Node.js).
• Windows runners: For Windows-specific builds and PowerShell/CMD scripts. Considerations:
• Platform compatibility: Choose a runner that supports your required tools and scripts.
• Pre-installed software: Review the available software to avoid unnecessary installation steps. For this guide, we'll use ubuntu-latest with bash scripts. |

Steps

Beyond the basic setup, templates in GitHub Actions offer a foundation for best practices and standards. This advantage is particularly significant for teams or individuals new to CI/CD or those transitioning to GitHub Actions from other systems. The templates can be easily customized and extended, allowing developers to adjust the workflows to fit their specific project needs while maintaining the integrity and efficiency of the initial setup.

Steps

In a GitHub Actions workflow, each task is organized into steps. These steps are detailed in the workflow file and can include various actions, such as running scripts or utilizing user-created actions available in the GitHub Marketplace.

Scripts within these steps can span multiple lines. The scripting language used depends on the operating system specified in the workflow's runs-on attribute. For instance, if you're using the ubuntu-latest runner, the default scripting language is Bash because it's based on Linux. However, you can use other scripting languages provided their interpreters are installed on the runner. Similarly, for runners using Windows, the default scripting language is CMD, though you can switch to PowerShell or others as needed.

What is "actions/checkout@v2"?

This is called an action, and can be written in many different programming languages, but usually TypeScript/JavaScript. Actions can do many things, such as installing software, changing configuration, downloading files, etc. This action automatically clones the branch associated with this pipeline. For more information on what this action does, visit its documentation page for options on how to configure it.

• Warning: actions can be authored by those other than GitHub. Be careful when referencing actions by their tag as this allows the developer to push any arbitrary code to that tag, which could cause security issues (i.e., they can run any arbitrary code in your repository.) Only use actions from those you trust.

• Be careful not to use too many actions (only when they are necessary), because they are difficult to run locally on your own computer because they use GitHub's Workflow Engine that, at the time of this writing, does not have the ability to be called from a desktop application. This means that it might be hard to run the action locally to see if it is correct and therefore developers will have a slow feedback loop.

• [GitHub - nektos/act: Run your GitHub Actions locally 🚀]{.underline} works for most actions.

• To debug your CI/CD pipelines effectively, consider setting up a temporary self-hosted GitHub agent. This allows you to run builds and inspect the application and build server outputs in detail. You can also integrate "sleep" steps into your workflow to pause execution at key points for thorough examination of the process and file system.

What is a Pipeline?

A pipeline is like a project manager for your build scripts. It orchestrates and automates the steps needed to build, test, and deploy your software.

Key Functions:

• Workflow Orchestration: Runs build scripts in a defined order across different environments (e.g., Windows, Linux).

• Parallel Execution: Improves efficiency by running tasks concurrently when possible.

• Status Monitoring: Provides insights into build progress, individual steps, and error troubleshooting.

• Build Server Management: Selects appropriate build servers for specific tasks.

Benefits of Pipelines:

• Increased Efficiency: Automates and streamlines the build process.

• Improved Reliability: Ensures consistent builds across environments.

• Enhanced Visibility: Provides clear insights into build status and errors.

• Faster Feedback Loop: Enables developers to quickly identify and fix issues.

Pipeline Runs:

• Each execution of a workflow is called a pipeline run.

• Provides insights into pipeline status and allows for cancellation if needed.

• Can be configured to send notifications to developers on failures or other events.

Pipeline Status:

• Green Pipeline: Indicates a successful build. However, ensure your build script is meaningful and actually verifies code quality.

• Red Pipeline: Signals a build failure. Investigate and fix the issue to unblock software delivery.

• Remember: A pipeline is only as good as the build scripts it runs. Ensure your scripts perform relevant tasks and tests to guarantee code quality.

[A Basic Continuous Integration Pipeline with GitHub Actions -- Burkhard Stubert (embeddeduse.com)]{.underline}

[Github Actions or Jenkins? Making the Right Choice for You | by Viduni Wickramarachchi | Bits and Pieces (bitsrc.io)]{.underline}

What is a build server?

A build server is a dedicated machine or cluster that automates the software build process, ensuring consistency and generating build artifacts for production. Triggered by events like code changes or pull requests, it clones the code, executes build scripts in an ephemeral environment, and provides a single point of truth for build results.

Build servers offer several advantages over local builds:

• Consistency: Eliminates discrepancies between developer environments.

• Reliability: Provides a stable and controlled build environment.

• Centralization: Acts as a central point of reference for build status.

• Build servers are typically disposable and replaceable, existing in pools, and can be hosted in the cloud or on-premise. They remain idle until triggered by a CI/CD system.

Helpful tips and best practices

• You might find it helpful to use an IDE Plugin, such as Github Workflows plugin in the IntelliJ IDEA IDE to author the workflow files. This is because the syntax can be fussy.

• Try to keep it to one command per step. It helps make the flow a bit more logical. Why can't I put everything in a single step? In theory, you could, but this would make it very difficult to know which step failed--you'd have to open up the step and check the logs. Notifications to stakeholders commonly include the failed step, so this is a useful debugging tool and helps you segment the logs for faster debugging. They're also needed for matrix builds, but we'll get to that later.

• Space out your steps by an empty line

• If the script starts getting long (i.e., more than a few lines), consider making it a separate script file, and then calling it in the runner.

• It's important to use the OS that you're developing on, because you have to be able to run those build scripts locally. So, if you have bash scripts on your CI server but can't run them locally (for whatever reason), then this means that the environments can't really be reproducible, because you have to ensure parity between the scripts on your computer and the CI. There are pros and cons though, as macOS cannot run cmd scripts for example.

• Remember that the dash means a new step. For example, the name key is prefixed with dash and this represents a new step.

• It's good to give steps names, otherwise they might not be clear what they're doing. Names are optional, however.

• It's very important that you keep this workflow formatted. Therefore, tools like yamllint are very useful. Poor indenting can make it super, super difficult to know what's wrong.

• If you're really stuck, look at a reference workflow file (that is properly formatted) to get your bearings.

• Normally, you'd want to set the runs-on to your development environment, assuming that that is the same environment where you are deploying to. If you need to run on multiple environments, you can use matrix builds (an advanced topic.)

Build notifications

Webhooks are a mechanism for one system to notify another system of events or updates in real-time. In the context of continuous integration (CI), webhooks are essential for facilitating automation and communication between various tools and services in the CI/CD pipeline. For example, if a build fails, then a webhook can be called, which can "send" a message to another service, such as Teams, Slack, and many others.

Webhooks are widely supported among many different integration providers.

• Build notifications are important because stakeholders must know if the build pipeline is failing, as it is the only route to deliver changes to production.

• Consider the audience for your build notifications. Normally, fixing a broken build is a shared team effort, so create a DL (Distribution List or a Distribution Group) or group with relevant team members. Avoid including individuals like the CEO.

• Set up immediate email/Slack notifications for pipeline failures through your CI/CD's integrations or webhooks.

• Note that not all pipelines require build notifications. Only those blocking the path to production, such as main branch pipelines, need them.

• Configure notifications to alert stakeholders impacted by broken builds using suitable channels like Teams, text messages, or emails, triggered only during failures.

Security

• Continuous Integration and Continuous Delivery (CI/CD) aim to streamline the development process by swiftly moving changes from a developer's environment to production. However, this rapid process can inadvertently introduce security risks, allowing malicious code to infiltrate production should an attacker gain access to a compromised account. This means that an attacker can easily push code to production--a two-edged sword. Therefore, having a good review system in place, along with 2FA (two factor authentication, requires that employees use their phone or other device to log in), dual-approval (two employees must approve of the changes before they go into production), dependency scanning, security scanning on code via SAST (static analysis), secret management, branch protection to limit who can and can't push to master, and YubiKeys can potentially limit or negate the damage done by attackers. Make sure to use proper identity management techniques by your provider, and don't share accounts.

• While CI/CD pipelines often run in isolated containers or virtual machines, this isolation isn't a bulletproof shield. Isolation prevents interference with other systems on the host, but it doesn't safeguard the contents within or shield them from potential internet threats. If, for instance, the CI/CD pipeline fetches a malicious resource, such as a malicious package, it could contaminate the build artifacts, propagating to customers, the production environment, or other artifacts.

• Moreover, CI/CD pipelines often possess secrets, usually in the form of environment variables or temporary files. If malicious scripts exploit these, they can access external resources by exporting the token, potentially racking up costs or jeopardizing sensitive data.

• Notably, hardcoding application credentials is risky. Even if they speed up prototyping, these hard coded secrets can be exposed, especially in open-source scenarios, leading to unauthorized access and potential misuse. And while storing API keys in a secure location might seem like a solution, at some point, these keys exist in plaintext, making them vulnerable.

• CI/CD is aimed at making it super easy to deploy to production, but not everyone should be deploying to production, mainly hackers. The answer isn't just about the choice between long-lived SSH keys or temporary tokens, as highlighted in the provided StackOverflow post. It's about a holistic approach to CI/CD security. Tools like YubiKeys provide an extra layer of security, but they aren't silver bullets. Physical devices, while helpful, can be lost or stolen. Thus, backup authentication methods and proactive monitoring are essential.

• Moreover, SMS-based two-factor authentication (2FA) isn't entirely secure due to risks of SIM swapping and SMS interception. In this realm, requiring multiple engineers to approve critical actions, leveraging platforms like Azure PIM, Google Cloud Identity, or AWS SSO, can add another layer of safety.

• When it comes to codebase and artifact access, only authorized individuals should have the rights. Furthermore, continuously monitoring the server side to ensure no unusual requests are made is pivotal. Secrets, API keys, or any form of authentication should be kept out of the codebase. Instead, leverage tools like KeyVault to store and access these secrets securely. Also, periodically run static security analysis tools to detect and rectify any exposed secrets in the codebase.

• Shifting left on security implies embedding security considerations from the start of the development process, rather than retrofitting them later. It's about ensuring that security is integrated from the onset and that reactive measures are minimized. After all, in the dynamic landscape of CI/CD, prevention is always better than cure.

Popular security static analysis tools

Open-Source Tools

• FindBugs with FindSecBugs Plugin: A static code analysis tool for Java that can identify security vulnerabilities with the FindSecBugs plugin.
• Checkmarx: Although primarily a commercial tool, Checkmarx does offer a limited free version that performs static code analysis for multiple languages.
• Bandit: Focuses on Python codebase and is designed to find common security issues.
• Brakeman: A static analysis tool for Ruby on Rails applications.
• SonarQube: Offers various language plugins and detects many types of vulnerabilities. The Community Edition is free.
• ESLint with Security Plugin: A widely-used linting tool for JavaScript that can also be used for security checks with the right set of plugins.
• Flawfinder: Scans C and C++.
• Cppcheck: Another static analysis tool for C/C++ codebases.
• YASCA (Yet Another Source Code Analyzer): Supports multiple languages including Java, C/C++, and HTML, but focuses primarily on web vulnerabilities.

Commercial Tools

• Checkmarx: A leading SAST tool that supports multiple programming languages and is designed for enterprise use.
• Veracode: Offers a static analysis service as part of a larger application security suite.
• Fortify Static Code Analyzer: Provided by Micro Focus, it covers multiple languages and offers integration with IDEs and CI/CD tools.
• IBM AppScan: Focuses on identifying vulnerabilities in web and mobile applications, supporting multiple programming languages.
• Kiuwan: Offers a broad range of language support and integrates with various IDEs and CI/CD tools.
• Synopsys Coverity: Supports multiple languages and offers CI/CD integration.

Integrating with External Services and Tools

Sometimes, your build pipeline might need to connect to other services because it doesn't have all of the information it needs inside of the repository. This is because there are external APIs, artifact repositories, secret managers, etc. that can't be part of the repository.

Why would you want to connect to external services, isn't everything I need in my repository? There are some things that can't be in your repository, because they are integrations, APIs, or managers that don't have a "physical" presence.

Security Reasons

Sensitive information, like API keys, database credentials, and other secrets, should never be stored directly in your repository. It's a security risk. Instead, these secrets are typically stored in specialized tools called secret managers (like HashiCorp's Vault, AWS Secrets Manager, or Azure Key Vault). When your pipeline needs to access a database or an external API, it will first fetch the necessary credentials from these managers. This ensures that sensitive information remains secure and doesn't accidentally get exposed or leaked.

Artifact Management

In many CI/CD pipelines, especially in large and complex projects, compiled code or built artifacts need to be stored or fetched. These artifacts are stored in artifact repositories like JFrog Artifactory or Nexus Repository. Connecting to these repositories can help fetch dependencies or store new ones post-build.

Integration and End-to-End Testing

Modern applications often rely on a myriad of microservices. When testing, it's crucial to ensure that these services work well together. For this, your pipeline might need to connect to service stubs, mocks, or even real services in a staging environment to perform integration or end-to-end tests.

Setup would depend on your CI software. You may need to connect a service to it.

After that, the secrets are normally available via environment variables in your pipeline. Typically, connecting to a service will pass along the inherited identity from the pipeline to the service, thereby authenticating you to it. Sometimes, you will need to manage these secrets manually.

Exercises

Set up a very simple pipeline. This pipeline should initially not be attached to PRs but instead run after a commit is merged. This is because there might be many mistakes while you set up the pipeline, and it might add an unnecessary blocker to those trying to merge. The pipeline should be as simple as possible and should just build the code and then run the simple test suite. Don't worry about publishing build artifacts, it should only build the code and return a status regarding if the build succeeded or failed. Make sure that the test suite runs and confirm in the logs that the test name and status show up correctly so as to diagnose any failing tests. Try to use a build template to build your application, and make sure that the template reflects the build script as closely as possible.

Set up the continuous integration server (or build server) to compile and run the code. Using the information derived from the planning stage, set up the build server to compile and build the code as a baseline. Developers will perform changes on the codebase, and should have sufficient tooling on their workstation to test the changes. This tooling should match what is run on the continuous integration system. It is important that developers have a stable copy on their workstation so that they can perform changes to the code because otherwise it would be overwritten by other developers' work. It is important that the tooling on the developer's machines matches the tooling on the build server because the build server's artifacts will be what is deployed. There should be sufficient tooling on the continuous integration system to make sure that there is a reasonable level of confidence that the changes are good. Choose activities that are prime for automation and are difficult for humans to do, such as compiling code, checking (tests), etc.

Continuously review and refine: Continuously review and refine the documented process. Encourage feedback from the team for improvements.

Programming a somewhat complicated weather application

This section explores CI/CD through the practical lens of building a sophisticated weather application. We'll dissect key features and demonstrate structuring a CI/CD pipeline using GitHub Actions.

Imagine wearing a project manager's hat and envisioning potential features for our weather application:

• Displaying precipitation data for user-specified locations

• Zoom functionality for map visualization

• Backend database for storing updated weather information

• REST API to serve data to the front-end

• Geolocation service to convert addresses to coordinates

• Caching mechanisms for performance optimization

• Historical precipitation data for a comprehensive user experience

• Pipeline feasibility for regenerating weather map tiles

Key Features and Development Strategy:

1. Interactive World Map: Our primary interface is a world map, designed to be interactive, allowing users to zoom into specific areas without reloading the entire map. We will be using the public open street map server for now but will show how you can self host it, including load balancing strategies.

2. Weather Forecast Integration: We will integrate real-time weather forecasting, beginning with current temperature displays at specific locations. This involves creating map overlays to show temperature variations across regions simultaneously.First, however, we're just going to get the temperature of our location.

3. Enhanced Map Visualization: The map will also display various weather parameters, such as wind speeds and temperature gradients. Given the potential for high user traffic, especially in densely populated areas like India, implementing load balancing and data compression strategies, such as vector tile maps, will be crucial.

5. Usage Analytics: Collecting data on user interactions with the map will provide insights to refine backend processes and enhance data visualization and user engagement on the platform.

Application Hosting and User Engagement:

• User features will include account creation and subscription to event forecasts. A backend batch job will manage notifications through a queuing system, supporting large-scale user notifications with email tracking.

[Interactive weather maps - OpenWeatherMap]{.underline}

Here's an overview of our application architecture.

Creating a new weather application using React involves several steps, from setting up your development environment to deploying the application. This book is not, of course, about how to learn React, so I won't be going into very much detail about how this React code actually works.

The first step is to provision a local development environment, enabling a quick feedback loop. This ensures immediate testing of changes, such as adding text to a weather application and seeing updates appear almost instantly in the React application.

You will have four different environments, each with a commonly used abbreviation. We will name some of the resource groups using these abbreviations as suffixes.

Environment Number Full Name Common Abbreviation

1 Local Development Local or Dev

2 Integration INT

3 Pre-production PPE

4 Production Prod

Naming conventions

There are a lot of things that you will need to name, such as pipelines, and other resources. Therefore, it is helpful to use a consistent naming scheme to make it easier to identify those resources.

[Define your naming convention - Cloud Adoption Framework | Microsoft Learn]{.underline}

Let's call the resource type CDP and the workload is our weather application.

It should therefore be prefixed with cdp-weather-web-prod

This provides a nice name we can use later and helps us inventory and group our resources, making it clear which resource is assigned to what.

Step 1: Set Up Your Development Environment

1. Install Node.js and npm:

• Visit [Node.js's website](https://nodejs.org/) and download the installer for your operating system. This will also install npm (Node Package Manager) which is essential for managing JavaScript packages.

• To verify the installation, run node -v and npm -v in your terminal or command prompt. This should display the current versions of Node.js and npm installed. Keep a note of this as you'll need it for later.

2. Install a Code Editor:

• A code editor will help you to write your code more efficiently. [Visual Studio Code](https://code.visualstudio.com/) is a popular choice among developers because it supports JavaScript and React out of the box, along with many useful extensions.

Installing Git

Windows:

1. Download the Installer:

• Visit the official Git website: [Git Downloads](https://git-scm.com/downloads).

• Click on "Windows" to download the latest version of Git for Windows.

2. Run the Installer:

• Once the download is complete, open the installer.

• Proceed through the installation wizard. You can accept the default settings, which are suitable for most users. However, you may choose to customize the components to install, the default editor for Git, and other options depending on your preference.

3. Verify Installation:

• Open Command Prompt (cmd) and type git \--version. This command will display the installed version of Git if the installation was successful.

macOS:

1. Install using Homebrew (recommended):

• First, install Homebrew by opening Terminal and running:

/bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"

• Once Homebrew is installed, install Git by typing:

brew install git

2. Verify Installation:

• In the Terminal, type git \--version to confirm that Git is installed.

Linux:

1. Install Git:

• Open a terminal.

• For Debian/Ubuntu based distributions, use:

sudo apt-get update

sudo apt-get install git

• For Fedora, use:

sudo dnf install git

• For other distributions, use the package manager accordingly.

2. Verify Installation:

• Type git \--version in the terminal to check the installed version.

Installing GitHub Desktop (optional)

Windows and macOS:

1. Download GitHub Desktop:

• Visit the GitHub Desktop download page: [GitHub Desktop](https://desktop.github.com/).

• Click on the download link for Windows or macOS, depending on your operating system.

2. Install GitHub Desktop:

• Windows:

• After downloading, run the GitHub Desktop setup file.

• Follow the installation instructions to complete the setup.

• macOS:

• Open the downloaded .dmg file and drag the GitHub Desktop application to your Applications folder.

3. Verify Installation:

• Open GitHub Desktop. The application should launch and prompt you to log in with your GitHub credentials.

4. Setup GitHub Desktop:

• After logging in, you can configure GitHub Desktop to connect with your GitHub repositories. You can clone existing repositories, create new ones, or add local repositories.

When you install Git, it typically comes with a tool called Git Credential Manager, which helps with authentication. If you're working in an interactive shell, you might see a pop-up from this tool when you try to access a repository. In a more basic command prompt environment, without a graphical interface, you'll need to follow specific instructions for accessing GitHub repositories. These instructions could involve pasting a link into a web browser or registering a device, using OAuth for authorization.

While you have the option to use a personal access token (PAT) for authentication, it's generally not recommended due to security risks, such as potential leaks and the extended lifespan of tokens. If you must use a PAT, consider setting its expiration to one week or less and arranging for it to be renewed periodically to enhance security.

First, ensure that you have cloned your GitHub repository to your local machine. Haven't made a repository yet? Then create one by creating a GitHub account and then creating a new repository, then cloning it locally.

Once you have the repository locally, create and switch to a new branch. You could name this branch something indicative of its purpose, such as "initial-commit" or "first-commit." Here's how you can do this using Git commands:

git checkout -b initial-commit

This command creates a new branch named "initial-commit" and checks it out, so you can start adding your changes to this branch. Do all of the following commands within the repository.

Run the following commands in that repository.

Step 2: Create a New React Application

Make sure that you have at least NPM 10.8.0 installed. You can update it by typing npm install -g npm@10.8.0 or whatever the latest version is.

1. Use Create React App:

• Open your terminal or command prompt.

• Run the following command to create a new React application named weather-app:

npx create-react-app weather-app

• This command sets up a new React project with all the necessary build configurations.

2. Navigate into your project directory:

• Change into the newly created project directory with cd weather-app.

Step 3: Run the React Application

• Inside the project directory, start the development server by running:

npm start

• This command runs the app in development mode. Open http://localhost:3000 to view it in the browser. The page will reload if you make edits.

Step 4: Integrate Weather Data

1. Choose a Weather API:

• For real-time weather data, you can use APIs like [OpenWeatherMap](https://openweathermap.org/) or [WeatherAPI](https://www.weatherapi.com/). You will need to sign up and obtain an API key.

2. Install Axios:

• While you can use the native fetch API, Axios makes it easier to perform API requests. Install Axios by running:

npm install axios

We need to access a weather API, but we're faced with a challenge regarding how to securely handle the API key. Storing the key directly in our code is not an option as it poses a security risk. If the key were to be leaked, it would be difficult to track and audit its usage.

To manage this securely for now, we will store the API key locally in a file named .env.local, which contains environment-specific data. Our React application will be configured to read from this .env file, allowing it to make calls to the API endpoint locally during development.

Later, we will explore solutions for safely using the API key in a production environment, ensuring it remains secure while accessible to the application.

Aside

Understanding the Build Process:

• Compiled Files: The files in the dist folder are the result of the compilation process. For example, if you're using a framework like React, the npm run build command transforms React code into plain JavaScript. This is necessary because browsers cannot interpret React code directly; they only understand JavaScript.

• Deployment Preparation: The dist folder contains the compiled version of your application, which is what you will deploy. This folder holds all the static files required to run your application on any standard web server.

Why Compilation Matters:

The compilation step is crucial because it translates code from development frameworks (like React) into a format that can be executed by a web browser, typically JavaScript, HTML, and CSS. This process ensures compatibility across different environments and optimizes performance for production.

Aside end

First create a .env.local file in the weather-app folder.

Replace the content of the dot env file with this.

REACT_APP_WEATHER_API_KEY=your_actual_api_key_here

Then make sure to add the .env.local file to your .gitignore file. Do not commit the .env.local file.

3. Create a Component to Fetch Weather Data:

• In the src folder, create a new file called Weather.js.

• Use Axios to fetch weather data from your chosen API and display it. Here's a simple example using OpenWeatherMap:

import React, { useState, useEffect } from \'react\';

import axios from \'axios\';

function Weather() {

const \[weather, setWeather\] = useState(null);

useEffect(() =\> {

const fetchWeather = async () =\> {

try {

const apiKey = process.env.REACT_APP_WEATHER_API_KEY;

const response = await axios.get(`http://api.openweathermap.org/data/2.5/weather?q=London&appid=\${apiKey}`);

setWeather(response.data);

} catch (error) {

console.error(\"Error fetching weather\", error);

}

};

fetchWeather();

}, \[\]);

return (

\<div\>

{weather ? (

\<div\>

\<h1\>{weather.name}\</h1\>

\<p\>Temperature: {weather.main.temp}°C\</p\>

\<p\>Condition: {weather.weather\[0\].description}\</p\>

\</div\>

) : (

\<p\>Loading weather\...\</p\>

)}

\</div\>

);

}

export default Weather;

Step 5: Include the Weather Component in Your App

• Open src/App.js.

• Import and use your Weather component:

import React from \'react\';

import Weather from \'./Weather\';

function App() {

return (

\<div className=\"App\"\>

\<header className=\"App-header\"\>

\<h1\>Weather App\</h1\>

\<Weather /\>

\</header\>

\</div\>

);

}

export default App;

You will then have to restart the application to pick up the changes in the .env.local file.

To test your application locally, begin by running the following commands in your terminal:

1. Build the Application:

npm run build

This command compiles your application and outputs the build files to the dist folder. Inside, you'll find several new files, including an index.html file, potentially some CSS files, and JavaScript files.

2. Start the Application:

npm run start

When you run the application, you should see that the API key has been successfully injected into the URL. In my case, since I didn't add my API key yet, there is an error.

Using the API key in production as we currently do is not ideal because it is exposed to the public. This exposure will lead to unauthorized use, resulting in significant charges or DDoS attack, meaning that our API quota will be exceeded. Fortunately, we're currently using a free version of the API, which limits the financial risk but not the operational risk; excessive fake requests could still deny legitimate users access.

Aside

Important Security Note Regarding GitHub:

When you commit an API key to a public GitHub repository, GitHub's secret scanning tool detects and invalidates exposed API keys for about 30 to 40 different providers within minutes. However, this window is sufficient for attackers to compromise your key before it's invalidated, leading to potential security breaches and loss of provider trust. It's crucial to never commit API keys to public repositories to avoid these risks. For more details on GitHub's secret scanning and best practices, you can refer to GitHub's documentation on secret scanning [About secret scanning - GitHub Docs]{.underline}

End aside

To securely store and manage these API keys, you can utilize Azure Key Vault. By integrating Azure Key Vault, you can inject API keys at runtime through custom endpoints, ensuring secure key management.

If you have an existing API consider using the Azure API Management Service. This service acts as a wrapper for existing APIs, adding valuable features such as authentication, rate limiting, quota management, and URL rewriting. In particular, we will leverage the URL rewriting capability to automatically append the API key and secret from the Key Vault to requests on our backend. This will hide the API key from the public URL and prevent it from being mis-used. Note that an attacker could still call our API multiple times to initiate a DDoS on our API Key, but we will get into rate limiting later.

Here's how to set this up using Azure API Management Service:

1. Create a New API Management Service: Begin by creating a new resource group, for instance, named 'CI-CD-Book-int' in the East US region. Name the resource as desired, such as 'My API Management Service', and fill in the organization name and administrative email according to your requirements. Choose the 'Developer' pricing tier.

2. Manage Identity: In the 'Manage Identity' tab, enable the system-assigned managed identity to allow your API Management Service access to the Azure Key Vault. This setup requires configuring Azure Role-Based Access Control (RBAC) rules to establish the necessary permissions.

3. Installation: Once all settings are configured, proceed to the 'Review + Install' tab and initiate the creation of your API Management Service.

5. Configure API and Testing: In the API Management Service:

• Go to 'APIs' and create a new HTTP API, such as a 'Get Weather'.

• The endpoint is just "/".

• Initially, use https://httpbin.org for testing to ensure the setup is correct.

• Select "Test" tab and then "Send". You should get a 200 OK response containing the content of the httpbin website homepage.

6. Key Injection and Endpoint Configuration: Adjust the backend settings to append the API key to incoming requests:

• Modify the service URL to http://httpbin.org/anything and save the changes.

In the below example, use the pretend API key listed below. This is because we are just testing our endpoint with a public server and we don't want to leak our actual API key.

Add the following policy to the inbound request:

\<!\--

- Policies are applied in the order they appear.

- Position \<base/\> inside a section to inherit policies from the outer scope.

- Comments within policies are not preserved.

--\>

\<!\-- Add policies as children to the \<inbound\>, \<outbound\>, \<backend\>, and \<on-error\> elements \--\>

\<policies\>

\<!\-- Throttle, authorize, validate, cache, or transform the requests \--\>

\<inbound\>

\<base /\>

\<set-backend-service base-url=\"https://httpbin.org/anything\" /\>

\<set-query-parameter name=\"api-key\" exists-action=\"override\"\>

\<value\>12345678901\</value\>

\</set-query-parameter\>

\</inbound\>

\<!\-- Control if and how the requests are forwarded to services \--\>

\<backend\>

\<base /\>

\</backend\>

\<!\-- Customize the responses \--\>

\<outbound\>

\<base /\>

\</outbound\>

\<!\-- Handle exceptions and customize error responses \--\>

\<on-error\>

\<base /\>

\</on-error\>

\</policies\>

At select save, then go back to the Test tab and then we run the request. You should get the following response or something very similar to it.

This is the expected response:

{

\"args\": {

\"api-key\": \"12345678901\"

},

\"data\": \"\",

\"files\": {},

\"form\": {},

\"headers\": {

\"Accept\": \"\*/\*\",

\"Accept-Encoding\": \"gzip,deflate,br,zstd\",

\"Accept-Language\": \"en-US,en;q=0.9,en-CA;q=0.8\",

\"Cache-Control\": \"no-cache, no-store\",

\"Host\": \"httpbin.org\",

\"Ocp-Apim-Subscription-Key\": \"986369bd5e1943aaac81cd4e87bde4f0\",

\"Referer\": \"https://apimanagement.hosting.portal.azure.net/\",

\"Sec-Ch-Ua\": \"\\\"Microsoft Edge\\\";v=\\\"125\\\",\\\"Chromium\\\";v=\\\"125\\\",\\\"Not.A/Brand\\\";v=\\\"24\\\"\",

\"Sec-Ch-Ua-Mobile\": \"?0\",

\"Sec-Ch-Ua-Platform\": \"\\\"Windows\\\"\",

\"Sec-Fetch-Dest\": \"empty\",

\"Sec-Fetch-Mode\": \"cors\",

\"Sec-Fetch-Site\": \"cross-site\",

\"X-Amzn-Trace-Id\": \"Root=1-66497521-4a028b2a52bd9d212f00e4db\"

},

\"json\": null,

\"method\": \"GET\",

\"origin\": \"154.5.165.200,13.91.254.72, 51.8.19.165\",

\"url\": \"https://httpbin.org/anything?api-key=12345678901\"

}

To ensure proper setup, start by creating a new Azure Key Vault and add a fake API key initially. This approach helps verify system functionality without exposing your real API key, especially since HttpBin is not secure for testing on a public website. Once you confirm that the system works as expected with the fake key, replace it with the actual API key. Additionally, update the endpoint to point to the actual weather API. Finally, conduct an end-to-end test by sending a sample request to see if everything is functioning correctly.

Here's how to do that.

Setting Up a New Azure Key Vault

1. Create the Key Vault:

• Navigate back to your resource group, specifically the CI-CD-Book-int one.

• Click on "Create New Azure Resource", search for "Key Vault", and select it.

• Name your Key Vault as "CI_CD_Book_KV" and leave the default settings intact.

• Proceed to create the vault by clicking on "View and Create", then "Create".

2. Configure Access Permissions:

• After creation, go to "Access Control (IAM)" on the left-hand side of the Key Vault.

• Click "Add Role Assignment", search for "Key Vault Administrator", and add yourself by selecting your user profile.

• Review and confirm the role assignment.

3. Manage Secrets:

• Once access is granted, navigate to the "Secrets" tab within the Key Vault.

• Click on "Generate or Import" to create a new secret. For instance, name it "weather-API-key" and set its value to "5934672295", then create the secret.

Integrating Key Vault with API Management Service

1. Link the Key Vault to API Management:

• In your API Management Service, locate the "Named Values" option under the subscriptions section.

• Add a new named value titled "weather-api-key" with the type set to "Key Vault".

• Select the "CICD Key Vault" and link the "weather-API-key" as the secret.

• Set the identity as the system assigned managed identity and save your changes.

• Confirm when prompted about adding the Key Vault secret User role to the IAM of this KV.

2. Update API Policy:

• Navigate to "APIs", select the "Weather API", and go to "Get Weather".

• Edit the policy using the policy editor. Insert the named value by typing "{{weather-api-key}}" into the appropriate field to dynamically insert the API key into API requests.

• Save your changes.

Now, update the policy to the following:

\<!\--

- Policies are applied in the order they appear.

- Position \<base/\> inside a section to inherit policies from the outer scope.

- Comments within policies are not preserved.

--\>

\<!\-- Add policies as children to the \<inbound\>, \<outbound\>, \<backend\>, and \<on-error\> elements \--\>

\<policies\>

\<!\-- Throttle, authorize, validate, cache, or transform the requests \--\>

\<inbound\>

\<base /\>

\<set-backend-service base-url=\"https://httpbin.org/anything\" /\>

\<set-query-parameter name=\"api-key\" exists-action=\"override\"\>

\<value\>{{weather-api-key}}\</value\>

\</set-query-parameter\>

\</inbound\>

\<!\-- Control if and how the requests are forwarded to services \--\>

\<backend\>

\<base /\>

\</backend\>

\<!\-- Customize the responses \--\>

\<outbound\>

\<base /\>

\</outbound\>

\<!\-- Handle exceptions and customize error responses \--\>

\<on-error\>

\<base /\>

\</on-error\>

\</policies\>

Now, you can use your base URI instead of calling the API directly. In my case, this is mine: [https://my-api-management-service.azure-api.net]{.underline}. In the React code, replace the call to the weather API endpoint with this URL.

You should be able to send a sample request to our API in the API Management service and you should be able to get a response back from the weather application.

After you've verified that everything is working, commit all changes and push to your branch.

If you're using a different cloud provider and don't have an API management service you can develop a custom application using C# or any other programming language of your choice. This application would consume the Key Vault at runtime through a managed identity. This method grants you greater control over the response processing and other aspects of API interaction because you are directly manipulating the code.

Tests

We are going to refactor the code a bit more to make it more modular. While it is possible to say that we are making it more testable, testing isn't a means to an end. Currently, the weather is loaded via useEffect. This is not very modular and couples the act of retrieving the weather with how it is rendered. If you want to change a single thing about the weather, then you have to change how it is displayed. This makes it difficult for multiple people to work on the application, as well as with feature flags because it is coupled to how it is displayed.

Let's do a small refactor and see how we can write some tests.

To write effective tests for the Weather component and to make the application more testable, we need to structure our code in a way that is easier to isolate and verify individual parts. Here are some improvements and test examples for the component:

Improving Code Structure for Testing

1. Decouple Data Fetching from Component Rendering:
   Extract the logic that fetches data from the API into a separate function or custom hook. This separation makes it easier to test the fetching logic independently from the component's rendering logic.

2. Use Environment Variables Judiciously:
   Ensure environment variables are used properly and securely, especially when building and testing. For production builds, consider server-side fetching or secure client-side API key handling mechanisms.

3. Error Handling:
   Add more robust error handling and loading state management to improve user experience and make testing these states easier.

Refactored Component Code

Here's an example of how you could refactor the Weather component to make it more testable:

import React, { useState, useEffect } from "react";
import axios from "axios";

// Data fetching logic extracted to a custom hook
function useWeather(apiKey) {
  const [weather, setWeather] = useState(null);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState(null);

  useEffect(() => {
    async function fetchWeather() {
      try {
        const response = await axios.get(
          `http://api.openweathermap.org/data/2.5/weather?q=London&appid=${apiKey}`
        );
        setWeather(response.data);
        setLoading(false);
      } catch (error) {
        setError(error);
        setLoading(false);
      }
    }

    fetchWeather();
  }, [apiKey]);

  return { weather, loading, error };
}

function Weather() {
  const apiKey = process.env.REACT_APP_WEATHER_API_KEY;
  const { weather, loading, error } = useWeather(apiKey);

  if (loading) return <p>Loading weather...</p>;
  if (error) return <p>Error fetching weather</p>;

  return (
    <div>
      <h1>{weather.name}</h1>
      <p>Temperature: {weather.main.temp}°C</p>
      <p>Condition: {weather.weather[0].description}</p>
    </div>
  );
}

export default Weather;

Writing Tests

Here are some test examples using Jest and React Testing Library:

import { render, screen, waitFor } from "@testing-library/react";
import axios from "axios";
import Weather from "./Weather";

jest.mock("axios");

describe("Weather Component", () => {
  test("renders weather data successfully", async () => {
    const mockWeatherData = {
      data: {
        name: "London",
        main: { temp: 15 },
        weather: [{ description: "Cloudy" }],
      },
    };

    axios.get.mockResolvedValue(mockWeatherData);

    render(<Weather />);

    await waitFor(() => expect(screen.getByText("London")).toBeInTheDocument());
    expect(screen.getByText("Temperature: 15°C")).toBeInTheDocument();
    expect(screen.getByText("Condition: Cloudy")).toBeInTheDocument();
  });

  test("shows loading initially", () => {
    render(<Weather />);
    expect(screen.getByText("Loading weather...")).toBeInTheDocument();
  });

  test("handles errors in fetching weather", async () => {
    axios.get.mockRejectedValue(new Error("Failed to fetch"));
    render(<Weather />);
    await waitFor(() =>
      expect(screen.getByText("Error fetching weather")).toBeInTheDocument()
    );
  });
});

Additional Considerations

• For production, consider implementing a backend service to handle API requests. This service can secure your API keys and manage the data before sending it to the frontend.
• Implement continuous integration (CI) to run these tests automatically when changes are made to the codebase.

If you were to run npm run test locally, you should see that all tests pass.

Now that we verified our changes locally, we can now set up the pipeline to see if those changes can be verified with a pipeline instead.

Making the Pipeline Build and Test Our Code

The current pipeline merely prints "hello world" and does not inspire confidence in the build artifacts. Let’s update it to perform meaningful tasks like installing dependencies, building, and testing the project. Edit your main YAML file with the following content:

name: Build client app

on:
  workflow_dispatch:
  pull_request:
    types: [opened]
  push:
    branches:
      - main # Triggers on pushes to the main branch. With workflow_dispatch, you can also run it manually.

jobs:
  build-and-deploy: # A single job to run everything for now.
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2 # Clones the repository.

      - name: Set up Node.js
        uses: actions/setup-node@v2 # Installs Node.js.
        with:
          node-version: "14" # Specify your Node.js version.

      - name: Install dependencies
        run: npm ci

      - name: Build the project
        run: npm run build

      - name: Test the project
        run: npm run test

      - name: Upload artifacts
        uses: actions/upload-artifact@master
        with:
          name: my-artifact
          path: path/to/artifact

Workflow Steps Explanation:

1. Checkout Step:
   Uses actions/checkout@v2 to clone your repository and set the working directory.

2. Node Version Setup:
   Sets up Node.js (version 14) for your build environment.

3. Dependency Installation:
   Installs project dependencies with npm ci.

4. Build and Test:
   Runs the build and test commands (npm run build and npm run test).

5. Artifact Upload:
   Uses actions/upload-artifact to preserve build artifacts (since the runner is ephemeral).

After updating, push the commit to your branch and create a pull request. The build will run, and you won’t be allowed to merge until the pipeline completes successfully.

Additional Considerations

Artifacts:
In the current setup, the build server is wiped clean after each run, which means build artifacts are lost unless explicitly saved. Use the actions/upload-artifact action to preserve these artifacts for later deployment or verification.

Note on Non-Compiled Projects:
Some projects (e.g., Python applications) might not generate traditional output files. In such cases, the source code itself (minus configuration files) may be considered the artifact.

Security Note:
When using actions from the GitHub Marketplace (e.g., actions/checkout@v2), be aware that version tags like "V2" are mutable. To reduce risk:

• Minimize reliance on public actions.
• Reference actions by specific commit hashes rather than mutable tags.

Accessing Artifacts Manually

To download artifacts from a GitHub repository:

1. Navigate to Your Repository: Log into GitHub and open your repository.
2. Access Actions: Click the "Actions" tab to view workflow runs.
3. Select a Workflow Run: Click on the desired run.
4. Find Artifacts: Scroll to the "Artifacts" section at the bottom of the page.
5. Download Artifacts: Click the artifact name to download a ZIP file containing the artifacts.

Deployment and Release Strategies

Deployment involves transferring build artifacts from the artifact repository to a customer-accessible server. In our workflow, after uploading the app artifacts, we can create a deployment pipeline. Note that deployment does not necessarily mean immediate customer visibility—a feature may be hidden behind feature flags.

Key Points:

• Stable Artifacts:
  Once created, do not rebuild artifacts. Rebuilding undermines confidence in the CI pipeline.

• Infrastructure as Code (IaC):
  Consider using tools like Azure Bicep templates for managing infrastructure. This approach is more manageable and scalable than manual portal setups.

Deployment Options

• Static Websites:
  For simple sites (HTML, CSS, JavaScript), using an Azure Storage Account and a Content Delivery Network (CDN) can be cost-effective and scalable.

• Server-Side Applications:
  For applications that require backend processing, consider Docker containers or other server infrastructures.

Setting Up Deployment with Azure

Below are the initial steps to deploy a static website using an Azure Storage Account and a CDN.

Step 1: Install Azure CLI

Ensure that the Azure CLI is installed on your computer.

Step 2: Log in to Azure

Open your terminal or command prompt and run:

az login

Follow the on-screen instructions to log in to your Azure account.

Step 3: Create a Storage Account

• Navigate to Storage Accounts: In the Azure portal, click on "Create a resource" and search for "Storage Account".

• Set Up Basic Details:

• Choose a subscription and select the existing resource group.

• Enter a unique name for your storage account.

• Select a region close to your target audience to minimize latency.

• Choose "Standard" performance, which is adequate for static content.

• Select "StorageV2 (general purpose v2)" for the account type, as it supports static website hosting.

• Review and Create: Review your settings and create the storage account.

Step 4: Enable Static Website Hosting

• Configure Static Website:

• After your storage account is created, go to its overview page.

• Navigate to the "Static website" settings under the "Data management" section.

• Enable static website hosting by selecting "Enabled".

• Specify "index.html" as the index document name and "404.html" as the error document path.

Make sure to disable storage account key access.This is important because Georgia county keys can be used to access your BLOB container from almost anywhere with.A weak form of authentication. This is just essentially a password. We're gonna be using something instead called a managed identity or a Federated credential.

In the Storage account, navigate to either the Access Management tab or the Access Control (IAM) tab. Add yourself as a Storage Account Contributor and a Storage Blob Data Contributor at the storage account level.

Sample HTML file with just some trivial contents, for example the text Hello world.

Example:

To upload an HTML file named index.html from your local machine to the '$web` container in your storage account, use:

az storage blob upload \--account-name cicdbookweb \--container-name '\$web' \--name index.html \--file /local/path/to/index.html

Step 7: Verify Upload

Confirm that your file has been successfully uploaded to the blob container:

az storage blob list \--container-name cicdbookweb \--output table

Step 8: Set Up Azure CDN for Faster Content Delivery

• Create a CDN Profile:

• Go to the Azure portal, click on "Create a resource", find and select "CDN".

• Choose or create a CDN profile and select a pricing tier (Standard Microsoft is recommended).To defaults and click.Next.

• Deployment: Note that it may take some time for the CDN to propagate globally.

Select review plus create.

Step 9: Access Your Deployed Site

• Site URL:

• Once the CDN is fully deployed, use the CDN endpoint URL to access your website, available in the CDN endpoint settings in the Azure portal.

• If you have configured a custom domain, use that URL instead.

You navigate to the host and the previous screenshot that you should see your sample HTML file.

Create a new workflow at .github/workflows/deploy.yml and insert the following content:

• Add a GitHub Actions workflow file to handle deployment:

name: Deploy to Azure Storage

on:

push:

branches:

- main

jobs:

download-and-deploy:

runs-on: ubuntu-latest

steps:

- name: Checkout Repository

uses: actions/checkout@v2

- name: Download Artifacts

uses: actions/download-artifact@v2

with:

name: your-artifact-name \# Specify your artifact name here

path: path/to/artifacts \# Specify path to download the artifacts to

- name: Deploy to Azure Blob Storage

uses: Azure/azure-cli@v1.1.0 \# Using official Azure CLI action for deployment

with:

inlineScript: \|

az storage blob upload-batch -s path/to/artifacts -d \$web \--connection-string \${{ secrets.AZURE_STORAGE_CONNECTION_STRING }}

az storage blob sync -s path/to/artifacts -d \$web \--connection-string \${{ secrets.AZURE_STORAGE_CONNECTION_STRING }}

env:

AZURE_STORAGE_CONNECTION_STRING: \${{ secrets.AZURE_STORAGE_CONNECTION_STRING }}

• Secure Your Workflow:

• Store your Azure storage connection string in GitHub Secrets to keep it secure.

Now if you go to the URL corresponding to the Azure CDN, potentially after 5 or 6 minutes for the CDN refresh, you should see your React application along with the weather app.

This approach works fairly well for simple projects, but it can become complex when managing multiple workflows. Currently, we have two distinct workflows: one that automatically deploys when changes are pushed to the main branch, and another that runs on pull requests to ensure confidence in the artifacts. The challenge arises in tracking which version or application is in production due to these separate workflows. It becomes unclear, for instance, after a pull request is merged into the main branch, which environment the application is currently in or if it's ready for QA.

To address this, GitHub offers features like jobs and environments that help structure workflows more clearly. These tools enable you to track your application's progression through its entire lifecycle in a visible and organized manner. This is crucial when multiple team members are committing to pull requests, potentially creating chaos without a clear order. Implementing structured workflows ensures you can easily identify which version is being tested and what is moving to the next stage.

Jobs and environments

Before we explore GitHub workflows, it's essential to understand the basics like jobs and environments. These elements are critical for structuring effective workflows, especially as we deploy our weather application. A clear grasp of these elements ensures that the workflow accurately reflects the application's current stage---whether it's integration, pre-production, or production. This clarity is vital for tracking feature releases and maintaining transparency about the status of ongoing changes.

Let's start with jobs.

Jobs in workflows are crucial for managing transitions between different stages and incorporating manual approval processes. Each job operates as an independent unit within a workflow, running on its own virtual machine or container and consisting of multiple steps. This independence allows for clear demarcation points where manual approvals can be inserted, with the ability to pause for up to 30 days without any ongoing computation.

Now, what are environments?

Environments in GitHub Actions enhance the deployment process by grouping jobs into distinct stages. This grouping not only helps in managing the deployment process more effectively but also conserves resources by pausing the workflow between stages, providing a unified and controlled view of the deployment pipeline.

Environments are particularly useful in complex workflows where multiple stages are involved, such as moving from integration to pre-production and then to production, facilitating a seamless transition and effective management throughout the deployment process.

Aside start

Workflow Structure and Naming:

• Use concise job names (ideally under 18 characters) for clear visibility in the GitHub interface.

• Structure workflows strategically to maximize parallelism. For example, separate build and deploy stages can run concurrently.

Example Workflow:

• Our workflow employs two jobs: "build" and "deploy". "Build" handles tasks like software compilation, while "deploy" manages security scans and deployment. Artifacts from "build" are passed to "deploy," ensuring isolated environments.

Efficient Deployment Strategies:

• Splitting workflows: Deploy to staging in one workflow, then trigger a separate workflow for production deployment after review.

• Creating separate jobs for each task can introduce overhead and complicate environment variable management by requiring broader scoping, potentially increasing security risks. It also involves repeatedly uploading and downloading artifacts, adding complexity. Additionally, while jobs can be parallelized, this may not always align with your script's structure. Organizing a script into multiple jobs can obscure the workflow's overall structure, making it difficult to understand dependencies and parallelization opportunities.

• Jobs allow for precise scoping of environments to specific tasks. For instance, if you have a production environment variable like a GitHub PAT, you can restrict its access to only the necessary steps. By assigning this variable to a particular job, such as deployment, you prevent unrelated jobs, like a "prepare cache" step that doesn't require production credentials, from accessing it. This ensures that production credentials are confined to the relevant job, enhancing security.

Aside end

Let's get this set up and show how you can use jobs and environments to create a pipeline to production, including manual approval stages.

jobs:
  build:
    runs-on: "ubuntu-latest"
    name: "Build" # this is optional
    steps:
      - name: "Checkout code"
        uses: "actions/checkout@v2"
      - name: "Install dependencies and build"
        run: |
          npm install
          npm run build

  test:
    runs-on: "ubuntu-latest"
    steps:
      - name: "Checkout code"
        uses: "actions/checkout@v2"
      - name: "Install dependencies and test"
        run: |
          npm install
          npm test

That workflow is displayed as follows.

No steps in the "deploy" job start until the "build" job is complete. This is because the "deploy" job has a "needs" on the build job.

There are few reasons why this is helpful. First, it is very clear from a glance which job(s) have succeeded and which were not executed. For example, if the build step failed, then the deploy wouldn't have succeeded. You can click on build to see its logs, just for its steps. The other reason is it is clear of the status of the deployment, and where the build currently exists. So, for example, say you do a deployment, then everyone can see that it is in the staging phase, and has not yet been deployed to production yet.

If we make a bunch of jobs, and a bunch of dependencies (e.g., needs), then it will be getting a bit more complex. When we start to create more complex workflows, jobs will be a way to help group related tasks together, and to provide them with dependencies.

• 

• [Revert "Only evaluate own String/Number/Math methods" · babel/babel@9ec1cb5 (github.com)]{.underline}

This workflow setup allows you to specify inputs and set the release type. For instance, if you wish to deploy commits from your main branch to the staging environment, you can manually input this, ensuring deployment stops at staging. Alternatively, you can deploy directly to production, though it will pass through each environment requiring manual approvals. You must configure these approvals and designate who has the authority to advance to the next step, such as requiring manager approval to move from staging to production.

You can select which environment you'd like as well.

run-name: Pipeline run by @${{ github.actor }} looks very useful for tracking release progress

If you try to start another instance of this workflow while it is running, then it will queue up and not run concurrently. This is important because otherwise you might have two scripts trying to deploy to production at the same time, which would cause a race condition (bad.)

You might want something like this,

• name: Display Release Version

run: echo "Deploying Release Version $RELEASE_VERSION"

Which can indicate which release is currently being deployed and where they are at.

You need GitHub Enterprise to set up pre-deployment checks (i.e., getting someone to approve before the next stage continues.) Since you are only allowed to specify five people per approval, then you can use teams (e.g., create a QA team, and a developer team) which should only use up two slots, but allow for more people to theoretically approve.

A typical scenario is to get QA to approve before it moves to the next stage. Let's show how to set up this sample scenario.

Step 1: Define Environments in Your Repository

First, you need to set up environments in your GitHub repository where you can specify protection rules including manual approvals.

1. Navigate to Your Repository Settings:

• Open your GitHub repository, go to "Settings" > "Environments" (found in the sidebar under "Security").

2. Create a New Environment:

• Click on "New environment".

• Name your environment (e.g., staging, production).

• Click "Configure environment".

3. Set Up Protection Rules:

• Under "Environment protection rules", you can add required reviewers who must approve deployments to this environment.

• Add the GitHub usernames of the individuals or teams who should approve deployments. For example, you can add a "QA team" that consists of a few people, and tey either all hae to approve or a single person has to approve.

• You can also specify other settings, such as wait timers or IP address restrictions if needed.

• Click "Save protection rules".

Step 2: Update Your GitHub Actions Workflow

After setting up your environments with required approvals, you need to modify your GitHub Actions workflow to use these environments.

1. Edit Your Workflow File:

• Go to your repository's .github/workflows directory.

• Open the YAML file for the workflow you want to add manual approvals to.

2. Add the Environment to Workflow Jobs:

• Identify the job(s) in your workflow that should require approval before they run. Add the environment key to those jobs, specifying the name of the environment you configured.

Here's an example snippet:

jobs:

deploy:

runs-on: ubuntu-latest

environment:

name: production

url: \${{ steps.deploy.outputs.url }} \# Optional: This can show a URL in the GitHub deployment

steps:

- name: Checkout code

uses: actions/checkout@v2

- name: Setup Node

uses: actions/setup-node@v2

with:

node-version: \'14\'

- name: Install dependencies

run: npm install

- name: Build and Deploy

id: deploy

run: \|

npm run build

echo \"::set-output name=url::http://example.com\" \# Simulated deployment output

Step 3: Commit and Push Changes

After editing your workflow file:

• Commit the changes: Provide a commit message that clearly states you've added environment protections with manual approvals.

• Push the commit to your branch.

Step 4: Trigger the Workflow

Push or merge a commit that triggers the modified workflow. If the workflow accesses a job that uses the protected environment:

• The job will pause, and GitHub will require the specified approvers to review and approve the run.

• Go to the "Actions" tab of your repository to see the pending approval.

Step 5: Approve the Workflow

• Authorized reviewers can go to the "Actions" tab, click on the workflow run, and then click "Review deployments".

• They can then approve or reject the deployment.

Step 6: Monitor the Deployment

After approval, watch the workflow continue its execution. If you provided an output URL in the environment configuration, GitHub would link the deployment to this URL for easy access.

Creating releases and "checkpoints"

In application development, a "release" marks the deployment stage where features become accessible to customers. This concept is crucial for tracking project progress, customer engagement, feature usage, and security updates. Releases also allow reverting to previous versions, though upgrading is generally preferred.

Managing releases can be complex. Determining version numbers, categorizing changes (major/minor), and meticulously documenting updates across files, documentation, what might break external libraries and customers, what changed and how to interpret those changes, and dependencies can be challenging. Manual processes are prone to errors, like forgetting version updates. This might also involve publishing your package to various repositories, all of which have their own requirements and the metadata must be correct to ensure compatibility with the developers.

GitHub Actions simplifies release management by tagging commits, auto-generating changelogs, and even refining commit messages into cohesive release notes, all within the deployment pipeline. You have to use one of the scripts or GitHub Actions below to do these releases as they do not occur automatically. Pre-release designations help track updates before production deployment.

There are two main ways to do versioning. One way is called SemVer (semantic versioning) which consists of a major, minor, and build number. This is commonly used when developing APIs and libraries, as developers should be made aware of breaking changes.

The other way is an evergreen strategy, which involves using a continuous version, for example the git hash or the date. This is usually used for consumer facing applications, like Teams or Skype. There are some exceptions, for example, when considering a major redesign, you might use a major version (e.g., Teams v2.) When was the last time you thought about Chrome's version number? It sort of auto-updates using an ever-green version strategy.

There's many different actions and libraries that you can use to create versions. It's recommended to use a pre-built solution as managing version numbers and incrementing them can become complex quickly.

Interesting: [Release Flow: How We Do Branching on the VSTS Team - Azure DevOps Blog (microsoft.com)]{.underline}

Here are just a few.

• [https://stackoverflow.com/a/69123272/220935]{.underline}

• https://github.com/GitTools/GitVersion

• https://github.com/conventional-changelog/standard-version

• https://github.com/semantic-release/semantic-release

• https://github.com/dotnet/Nerdbank.GitVersioning

• https://github.com/adamralph/minver

• https://github.com/conventional-changelog/conventional-changelog

• https://github.com/googleapis/release-please

• https://github.com/changesets/changesets

• [https://github.com/release-it/release-it]{.underline}

Setting up your repository: Build tools and more

Introduction

Clicking "Run" or "Start" in an IDE initiates a sequence of command-line tools that compile and manage dependencies to create build artifacts, simplifying the complex process with a single button. This abstraction can obscure the specific tools used, complicating tool selection for CI/CD pipelines.

What do I deliver to the customer, i.e., what are build artifacts? A typical software release often includes several components, tailored to the nature of the software and the target audience. Here are some of the common elements you might find:

1. Binaries: These are the compiled code files that are executable on the target platform(s). For desktop applications, these might be .exe files for Windows, .app packages for macOS, or binaries for Linux. For mobile applications, these would be .apk files for Android or .ipa files for iOS.
2. Libraries: If the software relies on specific libraries, these may either be bundled with the binaries or referenced as dependencies that need to be installed separately.
3. Documentation: This can include user manuals, release notes, and API documentation. Release notes are particularly important as they typically outline the new features, bug fixes, and known issues in that release.
4. Source Code (in some cases): For open-source software, the source code is often provided along with the binaries. Even in some proprietary contexts, source code may be provided to certain customers under specific agreements.
5. Installation Scripts/Programs: These are scripts or executable files that help users install the software on their system. This could include setup wizards for Windows, package installers for Linux, or dmg files for macOS.
6. Configuration Files: These files are used to configure the software for initial use, or to customize its operation. They might be in the form of XML, JSON, or other formats.
7. Database Files: If the application uses a database, the release might include database scripts to set up schemas or initial data sets.
8. License and/or Copyright Information: Legal documentation specifying the terms under which the software is provided.
9. Digital Signatures/Certificates: For security, the binaries and installer might be digitally signed to assure users that the software is genuine and has not been tampered with.
10. Additional Resources: This can include images, icons, data files, or other resources needed for the software to function correctly.
11. Patches/Updates: If the release is an update or patch to existing software, it may only include files that have been changed rather than the entire software package.

The contents of a software release can vary widely depending on the type of software, the platform it's being released on, and the policies of the developing company or organization. In enterprise environments, additional components like deployment guides, training materials, and support information may also be included.

The main artifact is the executable, or code, and are typically produced via your IDE. Sometimes, for manual build processes, there may be a team who is responsible for packaging the various build materials.

Visual Studio (for C++/C#)

• Build Commands
  Visual Studio uses msbuild to build projects. To see the exact commands:
  • Open the Tools menu.
  • Select Options.
  • Navigate to Projects and Solutions → Build and Run.
  • In the MSBuild project build output verbosity dropdown, choose Detailed or Diagnostic.
• Build Order
  The build order appears in the output window during a build (especially with verbosity set to Detailed or Normal).

Note: Build logs are primarily for troubleshooting. In legacy or complex projects, you might sometimes need to provide custom commands.

IntelliJ IDEA (for Java)

• Build Commands
  • The IDE uses its own builder. For Maven or Gradle builds:
    • Open the Terminal tab.
    • Run your build tool command (e.g., mvn compile for Maven).
    • The executed commands are printed in the terminal.
• Build Order
  • When using tools like Maven, the lifecycle phases determine the order. The order is also visible in the Build tool window messages.

Eclipse (for Java)

• Build Commands
  • Eclipse uses its internal builder. To view detailed build info:
    • Go to Window → Preferences.
    • Navigate to General → Workspace.
    • Enable Verbose output for the build.
• Build Order
  • Eclipse handles the build order internally. For more complex projects (often using Maven), the build lifecycle phases clarify the sequence.

Xcode (for C++/Swift/Objective-C)

• Build Commands
  • Open Xcode from the top menu.
  • Select Preferences and go to the Locations tab.
  • Set the Derived Data location to Relative.
  • After building, check the Report Navigator (rightmost tab) to view build logs.
• Build Order
  • The order is determined by your project dependencies and can be reviewed in the build logs in the Report Navigator.

Overall: Reviewing the output or log pane during builds is the best way to understand the commands executed and their sequence.

Build Tool Selection and CI Best Practices

When choosing build tools and configuring your CI pipeline, consider these guidelines:

• Favor Specific, Portable Tools Over Hacking
  A poor tool selection can lead to “CI bad smells.” Relying on custom shell scripts to patch issues may work initially but can later cause maintainability and portability problems. Instead, use established plugins and ensure tool versions do not conflict on your CI server.

• Avoid Out-of-the-Box Configurations
  Default configurations for external tools might not be optimal. Involve developers when defining quality gates instead of relying solely on customer requirements. This collaborative approach helps avoid irrelevant warnings and keeps the CI process efficient.

IDE Dependency and Portability Issues

Build scripts can become too tightly coupled with the IDE, leading to several problems:

• Hard-Coded Paths:
  Some IDEs install build tools in fixed locations. If your configuration references these paths, it can make your project IDE dependent, limiting portability.

• Configuration Challenges:
  Mixing personal IDE preferences with essential build settings can make collaboration difficult. Different environments (including CI servers) may not replicate the same configuration, leading to errors.

• Reproducibility on CI:
  Custom IDE settings, specific software versions, or environment variables injected at build time might not be available on CI. This discrepancy can change application behavior and hinder reliable builds.

Identifying Project Build Types

Determining the type of project and its build process can be done using a few heuristics:

• Use GitHub Linguist:
  Analyze the project’s primary languages. For example, if a project shows a high percentage of TypeScript and contains a package.json, it’s likely an npm project.

• Common Build Flows by Language:

  • Java: Code → Bytecode → Run on JVM.
  • Python: Code is interpreted.
  • C#: Code compiles into DLLs or EXE files.

• Check for Dependency Manifests:
  Look for files like package.json, Gemfile, pom.xml, etc., in the root directory. These files indicate the project type and guide you on how to build and test it.

Examples of Dependency Manifests and Build Commands

Below are several examples (from Heroku buildpacks) that illustrate how different project types are detected and built:

• Ruby

  • Files: Gemfile, Rakefile
  • Build: Not compiled in the traditional sense
  • Test: rake test
  • Heroku Buildpack Ruby

• JavaScript/TypeScript

  • Files: package.json
  • Build: npm ci or npm install (or corresponding Yarn commands; be cautious if both package-lock.json and yarn.lock exist)
  • Test: npm test
  • Heroku Buildpack Nodejs

• Clojure

  • Files: project.clj
  • Build: /bin/build or lien compile?
  • Test: lien test
  • Heroku Buildpack Clojure

• Python

  • Files: requirements.txt, setup.py, Pipfile
  • Build: Use pip to install dependencies
  • Test: python -m unittest (varies by project)
  • Heroku Buildpack Python

• Java (Maven)

  • Files: pom.xml (and related variants: pom.atom, pom.clj, etc.)
  • Build: mvn compile
  • Test: mvn test
  • Heroku Buildpack Java

• Java (Gradle)

  • Files: build.gradle, gradlew, build.gradle.kts, settings.gradle, settings.gradle.kts
  • Build: gradlew {check, test, build, etc.}
  • Test: gradlew test
  • Heroku Buildpack Gradle

• PHP

  • Files: index.php, composer.json
  • Build: composer install
  • Test: Varies depending on the application
  • Heroku Buildpack PHP

• Go

  • Files: go.mod, Gopkg.lock, Godeps/Godeps.json, vendor/vendor.json, glide.yaml
  • Build: go build
  • Test: go test
  • Heroku Buildpack Go

• C#

  • Files: .sln, .csproj, .fsproj, .vbproj
  • Build: Typically dotnet build
  • Test: Typically dotnet test

• C/C++

  • Files: Look for Makefile, CMakeLists.txt (for CMake), or .pro files (for qmake)
  • Build/Test: Depends on the build system (e.g., make, cmake, qmake)
  • Note: Makefiles can be used for various project types and might require inspection of the commands (gcc, g++, as, ld, etc.).

Typically, software development projects are complex and there may be different interpretations of what a project is. When organizing code, there are two main approaches: mono repo and multi repo.

Mono Repo:

• Advantages: Simplifies interdependency management, as all components are in one repository. Easier deployment and versioning together.

• Disadvantages: Git clone can become slow over time, though this can be mitigated by partial clones or Git VFS.

Multi Repo:

• Advantages: Each component has its own repository, allowing for independent deployment and versioning. This approach encourages creating public APIs for interaction.

• Disadvantages: Managing changes across many repositories can be complex, especially when multiple repositories need simultaneous updates.

Security:

• Multi repo offers better access control, as different repositories can have separate permissions.

Flexibility:

• Switching between mono repo and multi repo setups can be challenging and may disrupt Git history. Splitting a mono repo into multiple repos is generally easier than merging multiple repos into one.

If you're working with multiple developers, you may want to set up a GitHub organization to help manage multiple users access to your repositories. However, there are some security settings you should pay particular attention to. Below are the recommended settings when creating a new GitHub organization.

Setting up user accounts on GitHub

Setting up user accounts in GitHub Enterprise and ensuring secure access involves several steps. Here's a comprehensive guide to help you manage user accounts and enforce security measures like two-factor authentication (2FA) for accessing your GitHub repository.

Step 1: Create and Configure User Accounts

For GitHub Enterprise Server (Self-Hosted):

1. Login as an Administrator:

   • Sign in to your GitHub Enterprise Server as an administrator.

2. Navigate to the Admin Dashboard:

   • Click on the upper-right profile or organization icon, then select "Enterprise settings."

3. Manage Users:

   • Under the "Users" menu in the sidebar, click on "All users."
   • Here, you can add new users by clicking "Invite user" and entering their email addresses. Users will receive an invitation to join your GitHub Enterprise environment.

For GitHub Enterprise Cloud:

1. Organization Setup:
   • As an organization owner, go to your organization's page.
   • Click "People" and select "Invite member" to add new users by entering their GitHub usernames or their email addresses.

Step 2: Configure Permissions

1. Assign Roles and Teams:

   • Assign users to specific teams within your organization to manage repository access effectively.
   • Teams can be created from the "Teams" tab in your organization settings. After creating a team, you can manage repository access and permissions through the team settings.

2. Set Repository Permissions:

   • For each repository, you can specify who has read, write, or admin access. Navigate to the repository settings, click on "Collaborators & teams," and then add the teams or individuals with the appropriate access levels.

Step 3: Enforce Security Policies

1. Enable Two-Factor Authentication (2FA):
   • For enhanced security, enforce two-factor authentication for all users.
   • In GitHub Enterprise Cloud, go to your organization's settings, select "Security," then under "Authentication security," choose "Require two-factor authentication for everyone in your organization."
   • For GitHub Enterprise Server, navigate to the admin dashboard, select "Settings," find the "Authentication" section, and enforce 2FA by checking "Require two-factor authentication for all users."

Step 4: Secure Connections

1. Use HTTPS or SSH for Repository Access:
   • Ensure that all users access repositories using HTTPS or SSH.
   • Encourage users to set up SSH keys for a secure connection without needing to supply username and password each time. This can be done under their personal account settings by selecting "SSH and GPG keys" and adding a new SSH key.

Step 5: Audit and Compliance

1. Regular Audits:

   • Regularly audit user access and permissions to ensure compliance with your organization's policies.
   • Use the audit log feature to monitor activities. Access this in GitHub Enterprise Server by going to the admin dashboard and selecting "Audit log." For GitHub Enterprise Cloud, find it under your organization settings.

2. Continuous Training:

   • Continually educate users on security best practices, including the importance of strong passwords, recognizing phishing attacks, and securely managing their authentication credentials.

Additional Recommendations

• Review Third-Party Access: Regularly review and manage third-party application access from your organization's settings to ensure that only trusted applications have access to your data.
• IP Whitelisting: If using GitHub Enterprise Server, consider configuring IP allow lists to control which IP addresses are permitted to access your instance.

When you first set up your GitHub Actions enterprise repository, you may want to change a few things in the general actions permissions. First, allow "enterprise" and select "non-enterprise" actions and reusable workflows. Specifically, you want to only allow actions created by GitHub and disable the repository-level self-hosted runners. This is because allowing actions created by GitHub is technically a trusted source. However, if you allow actions from the marketplace, you must be careful about the creators you're using. If you only reference the version hash of that workflow file, arbitrary code could be executed. There have been instances where such code was not amenable to what you're trying to run in your repository. Additionally, avoid using self-hosted runners as they might allow someone to control or modify outputs for your runners, potentially injecting malicious code.

The next one is artifact and log retention. Set it to the maximum value—90 days in this case. This allows you to check for any malicious code that might have interrupted or interacted with your repository by reviewing the logs to see when a certain dependency was injected. It's also useful for debugging. For example, if you want to check if a build was contaminated with malware, or if you’re testing and need to determine which version was vulnerable to a security issue, retaining artifacts and logs is crucial. For an enterprise, it might also support auditing requirements.

For fork pull request workflows from outside collaborators, enable the "Require approval for all outside collaborators" option. This is crucial because you don't want workflows to run automatically when any collaborator forks your repository and makes a pull request. Without approval, those pull requests could contain malicious code that either consumes your repository resources (like for Bitcoin mining) or tries to access secrets. Although GitHub has improved security, there's still a risk of arbitrary code running on your runners. This is especially important if you're using self-hosted runners, where someone could execute arbitrary code. It's better to enforce these settings at the enterprise or repository level to avoid accidental modifications to workflow files, which could compromise your processes.

For member privileges under repository comments, check the box "Allow members to see comment authors' profile name" in private repositories. This helps identify who made a comment, providing more transparency and accountability within your team. It’s particularly useful when multiple contributors are involved, ensuring that feedback and discussions are attributed correctly.

Under Rules, there are several settings to check. First, require a pull request before merging. Set the required approvals to at least two, unless there's only one person on your team. This ensures at least one other person approves the pull request. Requiring a pull request before merging ensures that continuous integration runs, preventing random merges.

Next, dismiss stale pull request approvals when new commits are pushed. This is because an approval is based on the current state. If new code is pushed, it's a different pull request and needs re-approval.

Require a review from code owners to ensure certain parts of the repository get the proper review before updates.

Check "Require approval of the most recent reviewable push." This ties into the required approvals, ensuring each new push gets fresh approval.

Require conversation resolution before merging for auditing and to ensure all feedback is addressed.

Require status checks to pass to confirm continuous integration tests succeed.

Check "Require branches to be up-to-date before merging" to prevent issues when merging. However, this can create a bottleneck in large teams, as merging each pull request may take longer.

Under Rules, check "Block force pushes" to prevent rewriting history. This is crucial for auditing and ensures that others pulling the repository don't need to rebase unexpectedly.

You might also consider "Require workflows to pass before merging." However, it's wise to have a "break-glass" procedure for emergencies. For example, if your CI system is broken or you need to fix an urgent bug, bypassing checks can be necessary. This approach helps maintain operational flexibility while keeping security and stability in mind.

For repository roles, create a "break-glass" role used for emergencies only. Choose a role to inherit, which is write. Add permissions like "Jump to the front of the queue" under merge queue permissions, and "Request a solo merge." For repository permissions, allow bypassing branch protections. This role allows a member to elevate their permissions temporarily in emergencies. A repository administrator can assign this role, allowing them to bypass security checks once, ensuring break-glass procedures work as intended.

For two-factor authentication, ensure it's enabled. Check the box "Require two-factor authentication for everyone in the organization." This step greatly increases your organization's security.

Under Global Settings for Dependabot, you might want to check "Grouped security updates," though this depends on your preference. Also, enable Dependabot on actions runners to ensure it runs properly. If you have only self-hosted runners, check Dependabot on self-hosted runners to keep it in a trusted environment.

For secret scanning push protection, check "Add a resource link in the CLI and web UI when a commit is blocked." This provides helpful context and guidance for developers when they encounter blocked commits.

For third-party application access policy, set the policy to "Access Restricted" and allow people to create pending requests. This ensures that applications can't access your entire codebase without approval from the application administrator. This is crucial for security, as it prevents unauthorized access and ensures applications operate only with proper permissions.

Under Personal Access Tokens, ensure the option "Allow access via fine-grained personal access tokens" is checked. This provides only the necessary permissions for users and applications to access your repositories and organization. Also, set "Do not require administrator approval" for creating tokens to avoid hassle, especially since tokens can expire quickly. Additionally, disable or restrict access via classic personal access tokens, as they lack fine-grained control and can allow excessive permissions unless needed for legacy support.

Under Scheduled Reminders, you may want to connect your Slack workspace to notify developers when pull requests are ready for review. This integrates with your workflow, making it more convenient for developers to stay on top of reviews. You might also consider integrating with a webhook or another provider like email to ensure developers receive timely notifications and keep pull requests moving smoothly.

Under Repository Policies, set the base permissions to "Write" for all organization repositories. This ensures members have the lowest necessary access level, and higher permissions granted elsewhere will override it. For repository creation, set it to "Disabled" so members can't create their own repositories, enhancing security.

Disable repository forking to maintain a single source of truth and clear code control. Set outside collaborators to "Repository administrators allowed" to restrict who can invite external contributors.

Set the default branch name to "main." Restrict repository visibility changes to organization owners to prevent accidental exposure.

Disable repository deletion and transfer to maintain auditability and protect code history.

Under GitHub Codespaces, set it to "Disabled" unless you specifically want people to use it. GitHub Codespaces runs in a virtual machine outside your company's network, which can complicate auditing and security. It may also incur costs if developers leave Codespaces open for extended periods. Additionally, Codespaces might not meet your organization's data residency requirements.

Under Runners, set it to "Disabled for all organizations" to allow organizations to self-manage their own self-hosted runners. Avoid using self-hosted runners unless absolutely required, as they can be difficult to manage and keep up-to-date. They also run in an unsecure environment and operate on your company's network. It's better to keep everything isolated within GitHub. Allow self-hosted runners only if they need to access internal services that can't be run over the internet. Otherwise, disabling them prevents users from running self-hosted runners on personal devices, which could produce untrusted build outputs.

Create a new team called "Engineers" and potentially others like "QA." This avoids assigning permissions directly to each user. When a member leaves, you can remove them from the group, simplifying permission management. Assigning permissions at the team level makes auditing easier and ensures everyone in the group has the same access level.

You can also create a "Break Glass" team to temporarily elevate an engineer’s access for emergencies. Afterward, you can easily remove them, keeping access transparent and controlled.

When you set up two-factor authentication for your GitHub account, it's a good idea to set up a security key like a YubiKey. You probably won't want to use it for every commit, as it can be inconvenient to touch the YubiKey every time you commit. Also, install the GitHub mobile app for two-factor authentication. It's more secure than SMS codes and serves as a backup if you lose your phone or change numbers.

When creating a runner, typically, you would use the OS that most of your team members are using, or, the OS required to build the application

The instructions that github provides is for a stateful runner, much different from the runners cloud hosted by github. You will have to use kubernetes to re-create the nodes.

[Trace Context Level 3 (w3c.github.io)]{.underline}

A good dev setup guide (i.,e readme) should be clear and comprehensive. It should:

1. Describe the repository's purpose and fit within the organization.

2. Provide instructions on building, navigating, and using the repository.

3. Include links to wikis for setting up build tools.

4. Ensure the repository is self-contained, with all necessary dependencies easily accessible.

5. Specify contact information for the repository's owner or relevant team.

6. Include thorough documentation and possibly revise how wikis are managed on GitHub.

Here's a breakdown of what happened:

• Compromised Server: Attackers gained unauthorized access to one of Handbrake's download servers.

• Trojanized Software: They replaced the legitimate Handbrake application with a malicious version containing a Trojan (malware designed to disguise itself as legitimate software).

• User Downloads: Users who downloaded Handbrake from the compromised server unknowingly installed the Trojanized version on their machines.

• Remote Access and Data Theft: The Trojan gave attackers remote access to infected computers, potentially allowing them to steal sensitive data, install additional malware, or control the system.

How it Relates to Secure Delivery Mechanisms:

The Handbrake incident highlights several failures in their delivery mechanism:

1. Inadequate Server Security: The attackers were able to exploit vulnerabilities on the download server, indicating insufficient security hardening, patching, or intrusion detection measures.

2. Lack of Code Signing: Handbrake, at the time, didn't use code signing for their software releases. This means users had no way to cryptographically verify the authenticity of the downloaded file.

3. No Integrity Checks: The absence of checksums or hashes alongside downloads meant users couldn't easily detect that the file had been tampered with.

Lessons Learned:

The Handbrake breach underscores the importance of:

• Robust Server Security: Hardening servers, keeping software up to date, and implementing strong authentication and intrusion detection are crucial.

• Code Signing: Digitally signing software provides users with a reliable way to confirm the software's legitimacy.

• Integrity Verification: Providing checksums or hashes empowers users to independently check for file tampering.

• Security Awareness: Regularly remind users to download software only from official sources and to verify its integrity.

In Conclusion:

The Handbrake compromise was a costly and damaging incident that could have been prevented with stronger security measures in their delivery mechanism. It serves as a cautionary tale for all software developers and highlights the absolute necessity of prioritizing secure software delivery.

Comparison of Versioning Solutions

Tool Description Versioning Scheme Automation Language/Framework Pros Cons

GitVersion Derives semantic version based on Git history, branches, and tags. Semantic Versioning Build-time .NET, CLI Flexible configuration options, supports complex branching strategies. Can be complex to configure, requires understanding of Git branching strategies.

standard-version Automates version bumps, changelogs, and Git tags based on Conventional Commits. Semantic Versioning Commit-time, Release JavaScript Easy to use, enforces consistent commit messages. Less flexible configuration, requires adherence to Conventional Commits.

semantic-release Fully automates releases, changelog generation, and publishing based on Conventional Commits. Semantic Versioning Continuous Deployment JavaScript Highly automated, ensures consistent releases. Requires strong commitment to Continuous Deployment, can be challenging to set up initially.

Nerdbank.GitVersioning Embeds version metadata directly in the code using an MSBuild task. Semantic Versioning Build-time .NET Lightweight, good for simple projects. Limited configuration options, less flexible than GitVersion.

minver Infers semantic version from Git tags, supporting pre-release versions. Semantic Versioning Build-time .NET Minimal configuration, easy to get started. Limited control over versioning logic.

conventional-changelog Generates changelogs from commit messages formatted according to Conventional Commits. N/A N/A JavaScript Useful for generating changelogs independently from versioning. Requires adherence to Conventional Commits.

release-please Automates release PR creation based on Conventional Commits and labels. Semantic Versioning GitHub Actions JavaScript Streamlines the release process, integrates well with GitHub. Relies on GitHub Actions, requires adherence to Conventional Commits.

changesets Manages version bumps and changelogs for monorepos, using a separate file for change descriptions. Semantic Versioning Release JavaScript Good for managing complex monorepos, allows for granular versioning decisions. Requires additional steps for managing changesets, can be more complex for smaller projects.

release-it General-purpose release automation tool that supports various versioning schemes and plugins. Customizable Release JavaScript Highly customizable, supports various workflows and integrations. Can require more configuration compared to simpler tools.

Choosing the Right Tool:

• Complexity: For simpler projects with basic branching strategies, minver or Nerdbank.GitVersioning might be sufficient.

• Conventional Commits: If you are committed to using Conventional Commits, standard-version, semantic-release, or release-please are good choices.

• Continuous Deployment: For fully automated releases, semantic-release is the optimal choice.

• Monorepos: changesets is designed for managing versioning in monorepos.

• Flexibility: GitVersion and release-it offer high levels of customization and flexibility.

/// Start of Selection Consider your project's specific needs and your team's workflow to select the most appropriate versioning solution.

name: Deployment

on:
  workflow_dispatch:
    inputs:
      releaseType:
        type: environment
        required: true

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: false

jobs:
  create_release:
    permissions:
      id-token: write # This is required for requesting the JWT
      contents: write # This is required for actions/checkout
    runs-on: ubuntu-latest
    if: ${{ github.event.inputs.releaseType == 'production' }}
    outputs:
      release_id: ${{ steps.create_release.outputs.release_id }}

    steps:
      - name: Create Release
        id: create_release
        uses: actions/github-script@v6
        with:
          script: |
            const release = await github.rest.repos.createRelease({
              owner: context.repo.owner,
              repo: context.repo.repo,
              tag_name: `v${Date.now()}`, // This is an example tag format. Customize as needed.
              name: 'Production Release',
              body: 'New production release',
              draft: false,
              prerelease: false
            });
            return release.data.id;

  staging:
    needs: [create_release]
    runs-on: ubuntu-latest
    environment:
      name: staging
      url: https://github.com
    steps:
      - name: Check out repository
        uses: actions/checkout@v3

      - name: Deploy to Staging
        run: |
          echo "Pretending to deploy to staging environment"
          sleep 30

  production:
    needs: [staging]
    if: ${{ github.event.inputs.releaseType == 'production' }}
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://github.com
    steps:
      - name: Check out repository
        uses: actions/checkout@v3

      - name: Deploy to Production
        run: |
          echo "Deploying to production environment with release ID ${{ needs.create_release.outputs.release_id }}"
          sleep 30

Importance of versioning

• Versioning is necessary because it is important to keep track of which version of an application is published. This is important because when developers test the application, or need to find and report bugs, they must do so in a specific version. This allows the ability to reproduce the bug and to create a fix for it. Another version of an application might not have that bug, and so a developer would not be able to recreate that bug. Or, the version might have a different variant of the bug with different source code, and thus the fix for one version might differ substantially.

• It also allows documentation and marketing material to be associated with a specific version, or to keep track of which features were developed and when they were released. This is especially important for large, complex features where the work may have to be divided among multiple teams. If the teams are large and complex, then it might not be clear if the feature is done or how long it has existed in production.

• Another reason is for auditing and compliance. Imagine that there was a security issue in your application. Do you know how long the security issue has existed for? If you keep track of versions in your software, then you are able to go back in time and perform the same tests to check if the previous versions are vulnerable to those exploits.

• Depending on how your application is structured, you may have multiple versions of the application that you support.

• Ensure you version your application accurately. Proper versioning allows tracking and differentiation of multiple application versions. Whether addressing issues, making sales pitches, or managing deployments, unique version identifiers or tags are crucial. This prevents confusion among developers and stakeholders, ensuring everyone knows which version is in use, even amidst deployment challenges or staff absences.

• Versions allow you to.Make sense of the development process. So when application version is released, it's usually corresponding to some sort of.Some sort of tradition or segue or?Um.You know organizational thing that happens, so for example like the QA testers test this or something like this or.Getting the key testers.Test this version or the version is set for release and stakeholders are notified that this version is associated to this feature set associated to this thing and it allows for, you know, keeping track of it and associating what's actually being released with what's actually.Provide to customers.It's also good for morale as well. You know, you've released something and this is contained within this version which is, you know, associated to this and such and such.

Role of Dependency Managers

• Some package repositories can help manage your artifacts. The reason why you would want to have a certain package repository is to allow you to make sure that the artifacts that you are publishing are in the right place and are available to all developers. By having a central source of truth, this reduces confusion with (potentially) multiple versions referring to different copies of the application (which differ in the underlying code.)

• Artifacts are meant to be stored immutably. Git repos, theoretically can rewrite history, which is not a useful property for immutability.If you are, the reason why they artifacts are mutable is because they say if they weren't, say if you could change them, you put whatever you want them.Then the version of you know one point 2.3 doesn't correspond to.That code anymore.Yeah, it's one point 2.3 corresponding to whatever the changes were before you made it. So there's technically now they're becomes two version numbers.And you know, kind of defeats the point. You know, now you're putting it to a point of time which was immutable. So now you have to say like an extra long version number. So kind of treats the point of keeping them immutable and such.Yeah.I think also it can make things very confusing as well. You have version one point 2.3 and it's like, well technically which? Which one is that? Is that after modifications that before it is that? What is that? And even small changes in the source code can change the output as well, so.It is really complicated to know.Like.What's the impact of some of those changes? And it doesn't mean that you can never release a new version. It just means that.That old version that's insecure version, whatever you can lock away or whatever, but just don't have another application called that version essentially.That's kind of.how it works

• The dependency manager is software that normally runs on your computers and is usually specific to the type of application that you are trying to build. Its responsibilities are trying to determine the correct versions for your application to run. It may read a dependency manifest (sometimes called a package or lock file) that the programmer creates to indicate which versions of dependencies that your application needs.

• When your program needs certain versions of the dependency, then the dependency manager is able to retrieve those versions easily. This is because it has internal logic which is able to resolve the dependency tree and automatically download the right dependencies for use.

• A good dependency manager (or somewhere to host your artifacts) abstracts away most of this complexity by having a way to store your artifacts and provides instructions to your dependency manager (run locally) that can help resolve potential dependency conflicts. The artifact manager is just a server that hosts the artifacts, including metadata and might enforce ACLs.

• Imagine if distributing software was not standardized (in the past it was a bit more complex, but I will leave out the history.) This means that someone (usually a developer) has to read specific instructions on how to install the software, and to make sure that only that version is installed, and to reference it correctly. The issue with this approach is that it is tedious, error-prone, and time-consuming. The instructions provided by the package maintainer might be incomplete or may not support adding multiple versions of the software on the system. The package maintainer can't possibly know all of the software that is installed on the customer's computer and thus cannot know about all of the potential packaging conflicts. Trying to find out which versions of two different software are compatible with each other may require manually managing the versions and trying different versions. For example, "v1" only works with "v2" of application "B", but application "B"'s v2 is stored somewhere else, and might require application "C".

• When artifacts are published to an artifact server, the artifact server usually has specific functionality catered specifically for the distribution of artifacts. This may include allowing developers to connect to an artifact repository (which seamlessly integrates with other tools.) It may also be required for package managers which rely on a certain format for the package manifests and metadata to list available versions and to do dependency resolution. It also allows for finer-grained access control on the artifacts, whereas if everything is stored in a Git repository, it is likely that all of the files have the same access control.

• Package repositories can also allow you or have the capability to generate package manifests to allow other developers to easily consume your artifacts or packages as part of their build processes.So for example, let's just take the NPM registry. I guess that's probably considered a package repository or maybe the Maven one. So if you use the NPM repository.This is what you do is you can just, you know, run app, install whatever this package and it just grabs it for. You don't have to download it. You don't have to figure it out for any scripts or anything like that. You don't have to check to see which version of this package is supported with the other applications and other packages that you have.Saw this while.You don't care what versions of dependencies that that that package requires. It just automatically figures it out. And that's kind of.Part of the metadata supplied to the package repository. That just kind of does it for you. And if you want to update, you just update it and you know, package checks the package repository and.And does the update and such.This also is.Kind of different security as well, so I'll give this a little bit later. So given that package repositories are immutable, this means that.The securities guys were done and these packages and if there's a security issue found one of the dependencies for example, then since all dependencies are already classified and and associated to all of the packages.Then it becomes very clear which packages need to be upgraded, or are you know that or something like that. And this allows you to trace back all of your packages and see which ones are.Need to be upgraded? Which ones have security issues or which ones have malware, etcetera. And since you saw through the package repository, there's a link to that, so it's very easy to check.And this is an approach used by many.ah

• Package repositories usually have the capability of ensuring file integrity through the use of checksums. This can help ensure that the integrity of the package is safe from corruption and may also provide capabilities to host it on multiple servers or to provide backups. The act of backing it up is abstracted away using the artifact manager and allows you to focus on coding.

Version numbers: difference between internal builds, release builds, and customer builds

• What are internal builds? They are builds that are generated from the pipeline that aren't intended for release to customers. They might be generated many, many times a day (for example, as part of a branch or PR pipeline.) Normally, these builds aren't retained for long and are not released to customers.So theoretically, you know, an internal list could be promoted to an external release. Basically the only blessing that it gets essentially, I guess you caught that is it's been accepted or it's OK for stakeholders that moved to the next stage or something like that. So if you're using continuous deployment.Technically, your internal releases are likely going to be external releases. Normally, internal releases correspond to.Like temporary. So yeah, kind of that's what you calling internal builds. I would say internal builds are ones that are generated by the pipeline.Uh.That are intermediaries discarded while the developer pushes to that branch and have the pre open for example like after they push the code. Yeah, they're balls are immediately not useful because it's not the code that is.That's being reviewed so.But The thing is, is that.GitHub, well, you know, keep.Keep the skirt. So you may want to set the retention policy a lot lower for that.

• What are release builds? These are builds which are destined to be released to customers, but are not ready for customer consumption yet.So you may depending on your software or something like that, maybe something called the release pipeline, which is like a different user interface, but essentially it's the same same principles as like regular.As as a regular pipeline normally it's kind of more suited towards releases. For example, it takes in a set of artifacts.That have been published.By either deploy request pipeline or the main branch pipeline or something like that and then it goes on and can do some stuff in parallel. It's usually like.Green based approach or something like that and.The tasks are kind of set up.In like a different formats kind of more suited for.For releases so.And it allows you to pick and choose which.No artifact version that you want for.That's to be deployed.

Versioning Strategies

• It's good to have a clear versioning strategy (and clear versions), because if, for example, the versions were just a string of 32 letters (perhaps it is) but you had to use a spreadsheet to convert it into something else (i.e., a lookup table.) This would make it difficult in day-to-day operations because the act of having a version number is important for many reasons.

• Many people are interested in the status of the artifacts, albeit indirectly. For example, if the PM needs to know if "feature X" is being released, then they need to know where it is in the process and what the status is. Developers would also want to know so that they are able to meet the deadlines.

• There are some human aspects to versioning as well. When you are using a "manual" versioning strategy such as SemVer, it is possible that this is not added in code review. Therefore, it can be helpful to create PR templates which can remind people to make sure that a release is created. If not, then it is possible to push an empty commit which can "tag" a version in Git. This approach might be preferred if the releases are manual, and can help keep things consistent. In this case, the empty commit would purely be for the CI/CD pipeline to tag the associated commit, but the commit itself would have no data. It would simply refer to the tag (or the snapshot.) It is important to not make any changes to the code in this commit (except maybe the version number) because any small changes to the code (to generate the commit) might impact behavior, especially if it has already been tested. It also makes it unclear what changes are "fake" and which changes are real and can add noise to the commit log.

• The act of "cutting" a version or tagging can be a bit complicated because it requires the intersection of many processes. In this case, it can rely on build triggers, which versioning strategy to use, pushing multiple commits, and (embracing) a slow feedback loop with the CI runner, with lots of trial and error using unfamiliar syntax and complex branch regex rules. This requires that you know how all of these pieces fit together, and somewhat in depth, which can lead to difficulties trying to set up automated tagging in Git.

• Before making a release, a developer would have to manually type in a version number which would correspond to the SemVer that should be applied to this release. This depends on which type of versioning system you are using, however. Some versioning systems, such as incremental or evergreen versioning, use the date, which can be completely automated. The downsides with using the latter approach is that major changes can be introduced, and it is unclear to consumers that this has occurred. This is less of an issue when developing client-side applications, as customers are unlikely to care which version they are running. However, if you are creating a library or software that is intended to be used by other developers or a library, then backwards incompatible changes can cause breakage which can make it difficult for application consumers to consume the library and use it correctly. It is simply an act of communication.

• What would be an example of a poor versioning strategy?

  • Hypothetical example, the version number is a string of 64 characters, with 32 characters are just "a". This would make it difficult to distinguish between two different versions of the application, because you would have to visually inspect them and figure out how many "a"'s to ignore.

  • It would also be very long, and difficult to display. Some artifact managers or repository managers might not accept a version number that is that long.

  • Mixing commonly confused letters, like I, l, 1, and | together. This would make it visually difficult to distinguish two versions. It is possible to check if the versions are different using a diff-algorithm but this would be complicated.

  • Using special characters in the version number. The version number might appear in many places, and those places might not be able to accept special characters. For example, git tags and docker tags cannot accept certain types of special characters and have length restrictions.

  • Don't put private information in the version number. It is likely that it will be public in some way or another.

  • Make sure that the customer is able to view the version number, using a non-complicated method or procedure.

  • Trying to follow SemVer, but making too many exceptions. Therefore, it is important to make sure that they have the ability to be compared. Are they different? Is one greater than another one, or does it matter? Which one do we have to release for customers and which one is in testing?

  • You can change your versioning formats in the future if your needs change. But try to be consistent. Don't change it too often because this will cause a lot of confusion. Change it and be done with it.

• However, version numbers are a bit more flexible, but build number should only refer to a specific checksum or build of the application and should be immutable. For example, "iOS 17" refers to the latest copy of Apple's iOS operating system. This could be any versions from 17.0.1, 17.0.2, etc. and these are versions. Internally, they might have multiple builds per day that are not released to the public, or some developer beta versions.

• Versions in software are sort of like serial numbers for products. They allow traceability back all the way through the entire software development process, normally to find bugs or errors and for auditing purposes. It is also useful to know which version is deployed in production so that the product can be marketed correctly and developers know which version of the product contains bug(s), or the ability to know if a release was successful.

• Your versioning strategy should be able to trace the artifact back to the source code, and the versions of the build tools (optional but still useful.) Make sure that the environment is part of the version. There are several ways to version your application, depending on your type of application. SemVer is popular for libraries that have potentially API-breaking changes that consumers should know about. Consumers can specify (in their manifests) which versions of your library they choose to consume, and can do so safely because they know that SemVer will not be violated. In some cases, you might have an evergreen version of an application, or an application that is intended for the end-user (such as a website.) In this case, the API doesn't really have any breaking changes and SemVer might not apply. Therefore, consider using a date-based versioning strategy or a version that just increments. This will help you differentiate between releases.

• Do you segment your customers based on different platforms or levels of service? Make sure to include that in the version. For example, do you have a macOS application and a Windows one? Then make sure that the platform is in the version to make sure that they are differentiated. Is one intended for enterprise customers? Then add that.

Strategy Pros Cons

1. Semantic Versioning - Clear communication on changes - Requires discipline in adhering to the rules

(SemVer) - Popularized and widely adopted - May result in rapid version number inflation for unstable software

                          \- Differentiates between major, minor, and patch releases                       \- Not ideal for projects where public API doesn\'t change often but internals do

                          \- Easily integrated with dependency management tools

2. Date-based Versioning - Easily identifies when a release was made - Doesn't communicate the nature or impact of changes

(e.g., YYYY.MM.DD) - Neutral in terms of software changes -- it doesn't imply severity or size - Can be confusing if releases are made more than once a day

                          \- Can be combined with other versioning methods for clarity                     \- Not as widely adopted as other strategies

3. Sequential Versioning - Simple and straightforward - Doesn't provide insights into the nature or impact of changes

(e.g., 1, 2, 3...) - Continuously increments with each release - May give the impression of major changes even for minor updates

                          \- Users can easily identify newer versions

4. Release Trains - Predictable release schedule - Doesn't provide specifics about changes within each release

                          \- Can help ensure regular updates and feature drops                             \- Can lead to rushed or half-baked features if sticking strictly to the train schedule

                          \- Useful for larger organizations or projects with a lot of interdependencies   \- If a feature misses a \"train\", it might have to wait for the next scheduled release

Programming-language specific versioning strategy quirks

[7 Understanding Maven Version Numbers (oracle.com)]{.underline}

Storing artifacts and artifact retention

• It depends on the type of artifact. If this is an artifact that was distributed to customers, then in general, these are retained longer than artifacts created by CI as part of the build process (e.g., during a pipeline run) but were not released. This is because the artifacts created by the CI or CD pipeline (and are subsequently not released) can be created tens or hundreds of times a day, and it may not be worthwhile to keep them because they are created as an "artifact" of the build process and to show that the build process is still sane and are considered temporary files. Make sure that you only store the necessary files to build and run your application as part of its artifacts. This is because if you include too many files, then it can use up unnecessary space, and can be a potential security issue if it is unclear what those files contain (e.g., passwords, credentials, etc.) if they should not be shipped to customers. It is important to have a link to trace the inputs (source code) to the outputs (artifacts.) You can use Git tagging to create this link. This will allow more reproducibility later on, and can help fix issues (or to backport fixes) to prior versions should they have an issue. This depends on your versioning strategy, of course. For example, webapps that are evergreen do not normally have a version (for example, going to Facebook is always the latest version) but an enterprise desktop application might have many versions that have to be supported at one time.

• There are several things that would inhibit storing artifacts forever. One is the cost of storage, as depending on your application, artifacts may contain multiple dependencies and thus might be large.

• Some artifacts might be tricky to delete because if they are required as dependencies for other versions of software, then it can be difficult to untangle the dependencies. Therefore, keeping them for around longer is sometimes the safer approach.

• You may want to consider cold storage options if you have artifacts that are around for a while. This will allow you to save on storage costs.

• Not all artifacts have to be stored forever. Some are generated when a pipeline is run, and they are sometimes called "snapshots" or "revisions". These are usually temporary artifacts that provide the capability for them to be, in theory, published. In many cases they are never published and thus can be safely deleted. They should still be generated and retained for a while, however, because this will allow you to easily make a deployment should one be needed (or to revert to an older version of the software.)

<!-- -->

• Think about the utility of storing these files against the cons. If I store the entire node_modules folder, what do I gain that I don't have if I were to just store the revisions of the package.json and package-lock.json file? If NPM is down, consider using another registry instead of committing node_modules.

• It is difficult or not usually possible to delete items from the Git history. Artifact managers can deprecate or remove old version of the software or make them not available for package consumers.

• My pipeline runs a lot, why is that? Do I need to retain artifacts at each run?

  • A pipeline runs when a trigger has been hit (i.e., a PR was created, push to a branch, a new tag, etc.), or it was run manually. This is important because each run of the pipeline can generate artifacts, and the pipeline also should pass at various stages to prevent committing code that is not ready to be merged.

  • Commit to a Branch: When a commit is pushed to a branch, the pipeline can provide developers with early feedback. While this early-stage feedback is invaluable, it is most beneficial if developers actively utilize it. Typically, artifacts from these runs need not be stored long-term, as they often represent work-in-progress features.

  • PR Creation: It's imperative to run the pipeline when a PR is created and updated. This ensures the code meets the required checks before merging. Artifacts from this phase, much like the previous touchpoint, aren't usually stored long since multiple updates might be pushed before finalization. The pipeline must pass before the PR is merged.

  • New Tag Addition: If a new tag is pushed, it often signifies a deployment phase, and the pipeline might be geared towards initiating a release. In such cases, retaining the artifacts is crucial.

  • Post-PR Merge: The next time it might run is after the PR is merged. This appears strange, because it already ran on the PR, right? Things get a bit complicated and might depend on your CI/CD software's implementation.

    • When you create a PR, the CI/CD pipeline is run on (your branch, merged with the target branch.) This does not update your branch with the target branch.

    • If the pipeline was successful, the fact that it was successful might be retained for any length of time, usually 12 hours. This means that pushes to the target branch do not cause the PR pipeline to be re-run.

    • Since the target branch can update independently of the PR pipeline running, this means that there is a possibility for conflicting changes to occur (not merge conflicts.) This means that the pipeline has to be rerun on the target branch once it is merged to ensure that there are no issues. If, however, the target branch cannot be merged due to a merge conflict, then it will not allow the merge.

    • I'm assuming that this is the case because if there are multiple team members pushing to the pipeline at once, then this would quickly cause a bottleneck and cause all PRs to be recompiled every time there is a merge to the target branch. This would be fairly wasteful and would reduce throughput significantly. However, there is risk as there is a possibility of conflicting changes.

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | The provided document focuses on Maven's versioning system, SNAPSHOT versions, version range references, and how Oracle handles Maven version numbers. | | | | For many popular programming languages and build tools, documentation or guides similar to this can be found to help developers understand the intricacies of version management, dependency management, and related build tool features. Some examples include: | | | | 1. NPM (Node Package Manager) for JavaScript: | | | | - A guide on Semantic Versioning (SemVer) and how package versions work in NPM. | | | | - Explanation of version ranges, such as \^1.0.0 and \~1.0.0. | | | | - Details about package-lock.json and how NPM resolves dependencies. | | | | 2. pip for Python: | | | | - Details on how pip manages package versions. | | | | - Explanation of version specifiers like ==, \>=, \<=, etc. | | | | - Use of requirements.txt for specifying dependencies. | | | | 3. Gradle for Java (and other languages): | | | | - Information about declaring dependencies in Gradle and how it resolves conflicts. | | | | - Details on versioning, dynamic versions, and changing modules. | | | | - Explanation about the differences between implementation, api, compile, and other configurations. | | | | 4. RubyGems for Ruby: | | | | - Documentation about Semantic Versioning in the Ruby ecosystem. | | | | - Explanation of version specifiers in a Gemfile. | | | | - Details on how Bundler resolves gem dependencies. | | | | 5. Cargo for Rust: | | | | - Information on how Cargo handles Rust crate versions. | | | | - Explanation of Cargo.toml and Cargo.lock files. | | | | - Details on Semantic Versioning in the Rust ecosystem. | | | | 6. NuGet for .NET: | | | | - Guides on how NuGet manages package versions. | | | | - Details on versioning conventions and version constraints in .csproj files. | | | | - Explanation about packages.config and PackageReference. | | | | 7. SBT for Scala: | | | | - Details on library dependencies and how versions are resolved in Scala projects. | | | | - Explanation of versioning patterns and Semantic Versioning in the Scala ecosystem. | | | | For all these tools, the emphasis remains on helping developers understand version management to ensure consistent, repeatable builds and avoid the "works on my machine" problem. They discuss version constraints, resolutions, conflicts, and best practices. If you're interested in a particular language or tool, diving deep into its documentation will give you insights similar to what you've got for Maven. | +=============================================================================================================================================================================================================================================================================================================================================================================================================================+ +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

• Treat internal dependences as external dependencies. If you do an upgrade, and it breaks things (i.e., you're always on the latest and you do a PR to upgrade it, thus, running the CI pipeline), then just don't go through with the PR. Keeping it always on the latest version automatically means that previously passing code might start to fail and it might not be clear why.

• Another reason why you'd want to retain artifacts is because you want to have a reproducible build, even if you're generating a new one. This is because in order to know if something is broken (i.e., to narrow it down), then you have to have a stable environment to be able to rule out other factors. Also, dependencies can change functionalities, so if everything is always changing at once then it is difficult to test, and if you try to rollback changes (e.g., via a roll forward), it might not actually fix it because there are new dependencies that are being used instead of the old ones. There isn't an opportunity to test the dependency upgrades in isolation (e.g., via a PR.)

• Do not change artifacts that are in the artifact repository without changing their version number first.

• [12 From Build Automation to Continuous Integration (oracle.com)]{.underline}

• [TheNEXUS | A Community Project (sonatype.com)]{.underline}

• Changing the manifest of the artifact is important when you upgrade the version number so that it can be identified/stamped. Otherwise, it might theoretically be possible for two artifacts with different version numbers to be the same in terms of file content which would be weird, and it's also difficult for the customer to know which version they are running (or you.)

How do I version code and Docker images?

• A container is a managed execution environment that isolates its contents from the host, meaning the container doesn't know about other applications on the host. It shares the host's kernel and resources.

• A CI/CD server shares similarities with a container. It offers a stateless execution environment, often with some pre-installed dependencies. This environment is discarded post-run, ensuring a clean build environment every time.

• Once you've successfully built your program, you can test building it inside a Docker container, like "ubuntu-latest" for Linux builds. This mimics a CI/CD environment, which typically starts with a minimal setup, devoid of your application's specific dependencies or your codebase. You'll need to add these dependencies and your code to the container to build it.

• Note: When creating a tag for your CI/CD pipeline, you'll need to have a merged PR. Use git commit -m --allow-empty "Commit message here" to push a tag without any commit content.

• Note: if you are planning on using tags to support multiple versions of your software simultaneously, and are using trunk-based development, then this might be a bad idea. This is because tags only refer to a single commit which makes it difficult to change something at one point without changing everything after it. Therefore, you might be interested in different branching strategies. However, if the history is linear, and you're using a rolling versioning strategy (e.g., today's date), and the previous versions are never supported, then therefore tagging will provide a linear history, which should be suitable for most applications.

• All tags do is add an alias to a commit hash. It makes it easy to retrieve a particular version, as you can just view the tag and find the associated commit hash.

Git's git tag command lets you label specific commits. Here's how:

1. Lightweight Tags: A simple pointer to a specific commit.

git tag v1.0

2. Annotated Tags: These are full objects in Git's database. They contain metadata like the tagger's name, email, date, and a tagging message.

git tag -a v1.0 -m \"First stable release\"

• 3. Tagging Earlier Commits: To tag a non-recent commit, use the commit's hash.

• git tag v0.9 9fceb02

• 4. Pushing Tags to Remote: Explicitly push tags to a remote repo.

• git push origin v1.0

• Note: you may have to make a commit first (i.e., see previous command.) This is because some CI/CD software does not allow pushing tags because the only way to update the master branch is via a PR, and the PR must have at least one commit.

• 5. Deleting Tags: To remove a tag:

• git tag -d v1.0

• CI/CD tools may differ in their tagging setups. While Git allows for release tagging, some teams use third-party tools like Azure DevOps. If you need deep project management software integration, consider using built-in CI/CD offerings. Should you tag in Git? Weigh the benefits against potential confusion from mismatched tags and releases.

• Containerization:

• Docker packages software applications into deployable units called images. When running, these images are referred to as containers. With Docker, tags reference specific image versions.

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Setting up Static Analysis in GitHub Actions for a React App

Here’s a guide to setting up static analysis actions in your React app’s GitHub Actions workflow:

1. Define Your Workflow:

name: Static Analysis

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  static-analysis:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: "18" # Use your desired Node version

      - name: Install dependencies
        run: |
          npm install

      # Static analysis steps below

2. Choose Your Tools:

ESLint (Catches code style and potential errors):

- name: Run ESLint
  uses: actions/eslint-action@v3
  with:
    files: "src/**/*.js"
    eslint-path: "./node_modules/eslint" # Adapt if ESLint is installed globally

Prettier (Enforces consistent code formatting):

- name: Run Prettier
  uses: actions/prettier@v3
  with:
    files: "src/**/*.js"

Stylelint (Analyzes CSS and SCSS for style errors and inconsistencies):

- name: Run Stylelint
  uses: stylelint/actions/lint@v2
  with:
    configFile: ".stylelintrc.json" # Adjust config file path if necessary
    files: "src/**/*.{css,scss}"

SonarQube (Detects bugs, code smells, and security vulnerabilities):

- name: SonarQube Scan
  uses: sonarsource/sonarqube-scan-action@master
  env:
    SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} # Store your SonarQube token securely
  with:
    projectBaseDir: "."
    # Configure SonarQube project settings as needed

3. Customize Configuration:

• Create configuration files (e.g., .eslintrc.json, .prettierrc.json, .stylelintrc.json, sonar-project.properties) for each tool in your project root.
• Use eslint-config-react-app for a good starting point for React-specific ESLint rules.

4. Fail on Errors (Optional):

Configure actions to fail the workflow if issues are found. This enforces code quality. For example:

- name: Run ESLint
  uses: actions/eslint-action@v3
  with:
    files: "src/**/*.js"
    eslint-path: "./node_modules/eslint"
    failOnError: true

Security tools

Certainly! Here's a list of 10 security build tools, commonly known as static application security testing (SAST) tools or static code analysis tools. These tools scan your source code or binaries to find vulnerabilities without executing the program.

• dependabot to update deps

1. Checkmarx: A widely used tool that scans source code or even binaries to find security vulnerabilities.

2. SonarQube: A continuous inspection tool that provides insights on code quality. It includes a security analysis feature.

3. Fortify Static Code Analyzer (SCA): Provided by Micro Focus, it's a solution for detecting vulnerabilities in applications.

4. Veracode: A SaaS-based tool that provides full application scans, including static, dynamic, and software composition analysis.

5. Coverity: Offered by Synopsys, it provides static code analysis to detect and fix critical software defects in C, C++, Java, and more.

6. Klocwork: Used for identifying vulnerabilities; it integrates seamlessly into desktops, build tools, and CI servers.

7. RIPS: A PHP-specific static code analysis tool, known for its accuracy and speed.

8. Bandit: A tool designed to find common security issues in Python code.

9. Brakeman: A static analysis security vulnerability scanner specifically for Ruby on Rails applications.

10. GitLab Static Application Security Testing (SAST): Integrated into the GitLab CI/CD process, automatically scans the latest code changes for vulnerabilities.

It's important to note that no tool can identify all vulnerabilities, and they often need to be used in conjunction with other security practices—including dynamic analysis and manual code review—to ensure comprehensive application security.

Configuring build tools

Setting up the build tool with the appropriate configuration file(s)

• How do I force other developers to use extensions in the IDE? Workspaces, etc. You can do so implicitly by forcing these checks on the CI server, such as linting. It should have the same configuration as the ones that developers have. Therefore, developers will be inclined to use the recommended extensions because that way they won't have to redo a commit or a formatting change.

• IDE extensions such as linting on save/format, particular rules, etc.

• The testing commands won't work (usually) if you don't have any tests. It's advisable to write some more using CI/CD, or at least writing a single test, to show that the test build step can block the build if it fails.

• Try to run these commands on your local computer and see if they work (via the command line) outside of your IDE. This is because your IDE might have certain software pre-installed (or different environment variables set), which could make things more complex. If they work, then you're on the right track.

• git commit hooks? Might be too advanced, unsure.

Defining the build lifecycle, including phases and goals

• When you're building it, make sure that the runner's OS matches the ones that you're developing on. This is important because testing is done on the build machines, thus, if you build it on a different OS it is possible for idiosyncrasies. If you are using multiple OSes, do a matrix build, but don't use the artifacts from the other build steps (just test them.) Instead, just deploy the artifacts that are targeted for the destination OS. The latter part is optional but provides a baseline level of assurance that the tests/compile passes on the systems where it is being developed.

Getting better control on your dependencies, and lock files

• Lock files are a way to control your dependencies at a specific version, including any of its dependencies. Its goal is to make a reproducible build environment. The lock files are required because, even if you run the same commands in the same environment on non-dependency locked files, it's possible that it might use the latest or a different version of the dependency, depending on whatever dependency satisfies the one in the manifest and is in the artifact repository on the remote.

• You might already have a lockfile. If not, I would recommend that you create one. This is how you do it in the popular programming languages:

Sure, here's a list of instructions on how to utilize lock files during installation for each of the respective programming languages:

1. JavaScript (using npm):

   • Once you have package-lock.json in your project directory (e.g., from npm install).
   • Run:
     npm ci
   • This command uses the package-lock.json file to provide a clean, exact installation of your dependencies.

2. Python (using pipenv):

   • With both Pipfile and Pipfile.lock present in your project directory:
   • Run:
     pipenv install --ignore-pipfile
   • This ensures that the installation uses versions specified in the Pipfile.lock.

3. Java (using Maven):

   • While Maven's pom.xml doesn't function as a lock file in the traditional sense, you should specify exact versions in pom.xml.
   • Run:
     mvn clean install
   • Maven will fetch and install the exact versions defined in pom.xml.

4. Ruby (using Bundler):

   • Once you have a Gemfile.lock in your project:
   • Run:
     bundle install
   • Bundler will install the exact versions specified.

5. C# (using .NET Core/NuGet):

   • With specified versions in your .csproj file:
   • Run:
     dotnet restore
   • The .NET CLI will fetch and install the correct packages as listed in the .csproj.

It's crucial to ensure that the lock files (or their equivalents) are committed to your version control system so others can benefit from consistent builds.

How big should my PRs be?

• PRs should be concise and focused, encapsulating a single, coherent change or improvement that can be reviewed in context. Extremely small PRs are not helpful for providing context, and very large PRs can be overwhelming and difficult to revert.

• If a new feature isn't yet complete, use feature flags that allow partial functionality to be committed and tested without impacting users.

On pair programming

• Pair programming is a practice where two people work together, typically on the same machine in real time. It can be especially helpful if one is a senior dev mentoring a junior dev, providing a high-bandwidth communication channel and real-time feedback.

• Pair programming was popularized in Extreme Programming (XP). It is similar to code review, but occurs synchronously and can catch issues earlier. Async code reviews (via PRs) often complement or replace pair programming.

Note: Some projects (e.g., Python) don't produce traditional compiled outputs. Adjust your CI/CD pipeline accordingly.

Common Project Layouts (C#, Python, JavaScript/TypeScript, Java)

Below is a concise overview of typical project structures. Adapt them based on your specific needs.

C# Project Layout

• /src: Main source code.
  • ProjectName: Contains .cs files and ProjectName.csproj
• /tests: Unit and integration tests.
  • ProjectName.Tests.csproj
• /docs: Documentation.
• /lib: Libraries not in package managers.
• /tools: Build and related tools.
• /scripts: Build, deployment, and migration scripts.
• /packages: NuGet packages (less common in .NET Core).
• /.git: Git metadata.
• .gitignore & .gitattributes: Git configuration.
• /bin: Compiled binaries.
  • /bin/Debug
  • /bin/Release (usually for deployment)
• /obj: Intermediate files (not for deployment).

Python Project Layout

• /src (optional): Main source code.
  • your_package_name: .py files
• /tests: Unit tests (often using pytest).
• /docs: Documentation (e.g., Sphinx).
• /scripts: Utility or migration scripts.
• /data: Datasets or config files.
• /venv or /env: Virtual environment folder (ignored in .gitignore).
• setup.py: Packaging and distribution script.
• requirements.txt: Dependencies list.
• .gitignore: Git ignore rules.

Deployment often involves installing dependencies from requirements.txt using pip in a fresh environment.

JavaScript/TypeScript Project Layout

• /src: Main source code.
  • /components (for React, etc.)
  • /models or /types
  • /assets
  • /utils or /lib
• /dist or /build: Transpiled/compiled output (for deployment).
• /tests or **/**tests****: Unit/integration tests with Jest/Mocha, etc.
• /public: Static assets (index.html, CSS, etc.).
• /node_modules: Installed dependencies (ignored in Git).
• package.json: Project metadata and dependencies.
• package-lock.json or yarn.lock: Exact versions for deterministic builds.
• tsconfig.json: TypeScript compiler config (if using TS).
• .gitignore: Git ignore rules.
• .eslintrc / .prettierrc: Linter/formatter configs.

Java Project Layout

• /src: Main code/resources.
  • /main/java (source code)
  • /main/resources (config, images, etc.)
  • /test/java (test code)
  • /test/resources (test resources)
• /target or /build: Compiled artifacts (JARs, WARs).
• pom.xml (Maven) or build.gradle (Gradle).
• .gitignore: Ignore rules.
• README.md: Documentation.

Deployment and Release Strategies

Commonly used deployment strategies in CI/CD:

1. Blue-Green Deployment

   • Two environments: Blue (current production) and Green (new version). Switch traffic to Green when ready.
   • Advantages: Quick rollback, reduced downtime.
   • Disadvantages: Requires duplicated environments.

2. Canary Deployment

   • Gradual rollout to a subset of users before expanding to all.
   • Advantages: Early detection of issues, reduced risk.
   • Disadvantages: Requires sophisticated routing and monitoring.

3. Rolling Deployment

   • Incrementally replace old version instances with new.
   • Advantages: Simpler than Blue-Green in terms of environment duplication.
   • Disadvantages: Multiple versions run simultaneously during rollout, complicating rollback.

4. Feature Toggles (Feature Flags)

   • Deploy code behind flags; enable features when ready.
   • Advantages: Granular control, quick rollback without redeploy.
   • Disadvantages: Adds complexity if toggles are not well managed.

5. Shadow Deployment

   • New version runs alongside old in production, but real traffic doesn't affect live users.
   • Advantages: Test with real traffic without user impact.
   • Disadvantages: Resource-intensive, requires traffic mirroring setup.

The best strategy depends on your application, infrastructure, and team capabilities. Many organizations use a combination of these based on their needs.

Security and reproducibility

Add Git Credential Manager here too and say to use OAuth instead of PATs

Setting up security policies

• Security is very important when working with continuous integration repositories. This is because continuous integration has an achilles heel: it makes it very easy to get changes to production, which means that attackers can also make malicious changes to production easily. Therefore, it is important that you have strong security policies to make sure that only authorized users are able to access the repository and to perform certain actions. This means that one has to prevent a single unauthorized account from doing bad changes.

• Right now, each developer has an identity. This means that if a developer's account is compromised, then the damage can be tracked and the account can be disabled or reset.

• Make sure that a PR requires at least two approvers in order to be merged (not including the person who authored the PR.) These policies are usually managed by your CI/CD software. Of course, if there is only a single person on the team (or two people), then it might not make sense for two people to approve.

​// Start of Selection

Useful References

• Security best practices - Azure DevOps | Microsoft Learn
• Use Azure Key Vault secrets in GitLab CI/CD | GitLab
• Security hardening for GitHub Actions - GitHub Docs

After reviewing the information from Azure DevOps, GitLab, and GitHub Actions, we can combine the similar points and extract general themes as follows:

1. Authentication and Access Control

• User and Admin Access: Always grant the least required permissions. Use systems like Microsoft Entra Privileged Identity Management (PIM) for Azure or ID Tokens for OpenID Connect (OIDC) Authentication in GitLab for tighter access controls.

• Tokens and Service Accounts: Use tokens like the GITHUB_TOKEN in GitHub, service principals in Azure, and ID tokens in GitLab with specific scopes. Service accounts should have limited privileges and zero interactive sign-in rights. PATs (Personal Access Tokens) should be scoped, time-limited, and securely stored.

• Cross-repository Access and Service Connections: Scope access strictly to necessary resources, using service connections or authentication tokens, and avoid broad permissions.

2. Pipeline and Workflow Security

• Pipeline Construction and Execution: Use templates in Azure Pipelines, manage definitions with YAML, and enforce code review policies. In GitLab, make use of project-level secure files. Ensure jobs, like in GitHub Actions, run on specific branches only and sanitize inputs in build scripts.

• Runner Impact and Management: Understand the potential risks with compromised runners (e.g., in GitHub Actions). Utilize hardening measures, and if self-hosting runners, ensure they're isolated and grouped properly. Consider ephemeral runners for added security.

• Secret Management: Store secrets securely, using tools like Azure KeyVault, HashiCorp Vault in GitLab, or avoid logging them in pipeline variables. Use specific CI/CD variables for third-party integrations.

3. Third-Party Integrations and Tools

• Integration Best Practices: Disable outdated or insecure methods, like Azure Classic service connections or PAT-based GitHub authentication. When integrating GitHub with Azure DevOps, avoid personal accounts.

• Tooling and Automated Checks: Use tools like OpenSSF Scorecards in GitHub to detect supply chain risks, and consider tools for downloading secure files in GitLab or checking software bills of materials (SBOM) in GitHub runners.

4. Auditing and Monitoring

• Logging and Error Checks: Monitor service account activity, utilize security logs for account activities, and audit logs for organization activities (especially in platforms like GitHub). In GitLab, use error messages and administrative tools for diagnosing issues.

• Repository and Branch Monitoring: Implement repository-specific security policies, disable potentially insecure features like repository forking in Azure, and monitor production build definitions for potential credential leaks.

5. Deployment and Service Configuration

• Service Connection Scope: In Azure DevOps, specifically scope service connections to necessary resources. Avoid generic contributor rights and use purpose-specific team accounts.

• Environment Distinction: Maintain a clear boundary between test environments and production. Ensure production secrets are kept separate and securely stored.

By consolidating similar points across the three platforms (Azure DevOps, GitLab, and GitHub Actions), these general themes provide an overarching perspective on best security practices in DevOps environments.

The integration of security into the development and deployment lifecycle is an essential part of modern software practices. The term "DevSecOps" has emerged to emphasize this integration, underscoring the importance of security throughout the DevOps lifecycle. When it comes to books about CI/CD and even broader software development topics, there are several security-related areas that are often underemphasized or overlooked:

1. Shift-Left Security
   The idea behind "shifting security left" is to integrate security considerations and checks earlier in the development process. While some CI/CD books might touch on automated testing or code quality checks, they may not delve into integrating security scanning tools, static analysis for security vulnerabilities, or dependency checks in the early stages of the pipeline.

2. Secrets Management
   Handling API keys, database credentials, certificates, and other secrets is critical in CI/CD pipelines. Many books might not detail best practices for secrets management, like using secret managers (e.g., HashiCorp Vault, AWS Secrets Manager) or how to rotate and revoke secrets.

3. Immutable Infrastructure
   The concept of immutability, where infrastructure is never modified after it's deployed (and is instead replaced), can enhance security. This approach reduces the attack surface and ensures consistency. Many books might not discuss the security benefits of this method in detail.

4. Container Security
   As containerized applications and microservices architectures become more popular, securing containers is paramount. This includes scanning container images for vulnerabilities, ensuring containers run with minimal permissions, and using trusted base images. Many books might not go into the intricacies of container security.

5. Infrastructure as Code (IaC) Security
   IaC tools like Terraform or CloudFormation have their vulnerabilities. Some books might not discuss how to secure IaC scripts, best practices for code reviews, or the importance of scanning IaC for misconfigurations.

6. Runtime Application Self-Protection (RASP)
   RASP solutions provide real-time application security, detecting and blocking attacks in real-time. The integration of RASP into CI/CD might be an overlooked topic in many beginner books.

7. DAST and SAST
   Dynamic Application Security Testing (DAST) and Static Application Security Testing (SAST) are methodologies for identifying vulnerabilities in running applications and source code, respectively. Their integration into CI/CD pipelines can be crucial but might not be thoroughly covered.

8. Incident Response in CI/CD
   How to handle security incidents, particularly in a CI/CD context (like rolling back insecure deployments or patching in a CI/CD model), can be a topic that's glossed over.

9. Supply Chain Attacks
   Ensuring the integrity of software components, packages, and dependencies is critical to prevent supply chain attacks. Some books might not delve into the importance of verifying component integrity or the risks of using outdated or compromised packages.

10. Compliance and Auditing
    In regulated industries, compliance with security standards is mandatory. How to ensure and validate compliance in a CI/CD model might not always be explored in depth.

Given the importance of security in today's software landscape, those interested in CI/CD should seek out resources that give due attention to security considerations. If a general CI/CD book doesn't cover security in depth, consider complementing it with resources specifically focused on DevSecOps and security best practices in the context of modern software development and deployment.

• [https://stackoverflow.com/a/49552383/220935]{.underline} for using the job token instead of long-lived ssh keys to clone repo

• [Automatic token authentication - GitHub Docs]{.underline} github auth token, don't use ssh keys pls

Why is security important?

• Why care about CI/CD security? Isn't it all containerized and isolated anyway?

• The goal of CI/CD is about making it super easy to move changes from a developer's workstation and into production as quickly as possible. This means that malicious code can also make its way to production easily.

• The issue is that isolation can only go so far. Isolation is between other containers or VMs on the host, but does not protect the contents inside of the VM. This means that unauthorized users may run code or have it added to the build artifacts. It also does not prevent isolation from the internet, that is, if the pipeline downloads a malicious resource, then it can run it in the pipeline, and infect the build artifacts.

• Unauthorized users could create branches that contain malicious code that is run on the CI. This could cause intellectual property to be leaked, such as the contents of the source code on the CI pipeline being uploaded to another server. This is because it is usually possible to change the build script.

• This is of special importance because the pipeline usually contains secrets, injected through environment variables or added to temporary files in the machine. Malicious scripts may gain access to these credentials and can provision other VMs and resources outside of the pipeline. For example, using API keys.

• The exact process to set up YubiKeys and other devices varies depending on the device used, therefore, the instructions won't be provided in great detail here. However, I will go over an outline. The general thread is that engineers are issued a physical device that they can use to log in. The device does not usually store passwords and should not be considered a password manager. Engineers should contact the server administrator immediately if they suspect that their device has been lost or stolen. YubiKeys (and other devices) can't protect against every single possible threat, in fact, nothing can. Make sure that you have backup authentication methods in case your YubiKey is not available or it is lost (and you need to recover your account via the administrator.) YubiKeys can also be used on an individual level too to increase your security.

• How do I secure production? I already have 2FA set up and I get an SMS. Why is that considered insecure, I never let my phone out of my sight, right? Go into SMS attacks/SIM attacks.

• How do I make it so that another engineer has to approve someone else's request to access production systems outside of the CD pipeline? All major cloud providers support this scenario. It might be called "Azure Privileged Identity Management (PIM)", "Google Cloud Identity Platform", or "AWS SSO". For other cloud providers, see their RBAC (Role Based Access Control) or permissions pages to see if they support this scenario.

• Start with security when you start with your stories/tasks, don't re-engineer after-the-fact. Add tests for security.

• Only authorized users should have access to the codebase to download artifacts.

• Security monitoring on server side, making sure that application does not make strange HTTP requests.

• Keep secrets out of codebase

• SBOMs (secure bill of materials)

• Many application security scanning tools

• Secret scanning

• There might be unintentional security vulnerabilities in your application, such as embedding passwords inside of the build artifacts that are then published to customers.

• Packages used on CI servers may have security vulnerabilities, some which may have not been reported yet. Some genuine packages may have security vulnerabilities.

• This has happened on numerous occasions. For example, the removed packages from npm.

• Malicious use of resources. Even if nothing infects the build artifacts, or remains undetected, it can use excessive CPU and make the builds take longer. This can cost the organization money and resources.

• Expose security telemetry to developers ([Pragmatic Pipeline Security - James Wickett - YouTube]{.underline}) for example seeing attacks and being able to mitigate them if they are surfaced

• [OWASP Top 10 CI/CD Security Risks | OWASP Foundation]{.underline}

  • CICD-SEC-1: attackers can push code to branch(es) (or artifact(s) that are used in production) that are then (auto-)merged and deployed to production.

  • CICD-SEC-2: production identities are not secured, or production lacks the right ACLs and gives too many people access.

  • CICD-SEC-3: a dependency is compromised in a package by an attacker publishing a malicious version or by typojacking a legitimate dependency.

  • CICD-SEC-4: changing the build script causes the pipeline to re-run, thus executing the attacker's code in the build script.

  • CICD-SEC-5: pipelines are highly privileged and usually contain many secrets/passwords/access to production systems; malicious code can take advantage of this if it executes on the pipeline runner.

  • CICD-SEC-6: credentials printed to logs, embedded in code, embedded in images, or are unrotated.

  • CICD-SEC-7: generic advice on securing CI/CD architecture.

  • CICD-SEC-8: 3rd party apps linked to your account are too permissive and request too many permissions.

  • CICD-SEC-9: make sure to check checksums of things downloaded, sign your code and artifacts.

  • CICD-SEC-10: make sure to have sufficient logging and auditing set up.

• Sandboxed does not mean intellectual property is safe

  • Anything can be uploaded to a server in the post-install scripts, how to check which packages have these scripts (or scripts in general) npm view js-common --json | jq .scripts

  • Use example of ransomware npm package as an example

  • [https://stackoverflow.com/a/68865402/220935]{.underline} can help identify which package's install scripts are causing issues (disable each type of script, re-run build and verify)

• [FOSDEM 2023 - The 7 key ingredients of a great SBOM (archive.org)]{.underline}

• Best practices, such as not including API keys in your CI pipeline because they can be slowly dispersed throughout the organization and shared amongst multiple people, and might not obey the same security policies as the rest of the code.

• Static code analysis for security issues, as well as malware within the pipeline itself (e.g., malicious packages published to npm.)

• Make sure that only authorized people have access to the pipeline and can push stuff to it

• If running an open-source project, make sure that people can't submit PRs and run cryptominers and such

• Make the runners stateless to prevent secrets from being written to disk for long periods

• Don't print passwords to the console or API keys as those are written to the logs (which are stored for a long time)

• Signing the application does not mean that the build is reproducible. The fact that the build is reproducible is different from signing, because you can sign malware.

• Authentication data is hardcoded (in clear) under VCS (BP29)

  • " "Authentication data is hardcoded (in clear) under VCS" bad smell (BP29), is considered still relevant by our survey participants since it is mainly related to security issues"

  • Hardcoded application credentials are a security risk, because these credentials can be read by other developers. These credentials will then exist unencrypted on harddrives, meaning they can be part of backups, spread, or be read by other programs on the machine. Developers may use the credentials to do testing, even though those credentials should not be used for that purpose.

  • If working on an open-source application, credentials will be immediately taken advantage of if they are pushed to a public repository. While there are security scanners that usually revoke the credentials, this in and of itself makes it an inconvenience.

  • It also makes the application unportable because the credentials must be manually changed.

  • Counterpoints:

    • Rapid prototyping or test applications can hardcode their credentials, because setting up the necessary infrastructure can be time consuming. It may also require service connections and other boilerplate work that reduce the velocity.

/// Start of Selection

• Sabotage: Code added to popular NPM package wiped files in Russia and Belarus | Ars Technica

Malicious Code in Open-Source Software Targets Russia and Belarus

Overview

A developer inserted malicious code into the popular open-source package, node-ipc, targeting computers in Russia and Belarus. This act stirred controversy within the open-source community and raised concerns regarding the safety of free software.

Key Points

1. The Software Affected

   • The software, node-ipc, enhances other open-source code libraries with remote interprocess communication and neural networking capabilities.
   • As a dependency, node-ipc is automatically downloaded and integrated into other libraries, such as Vue.js CLI, which receives over 1 million weekly downloads.

2. Malicious Action

   • The author of node-ipc introduced a version of the library that identified and sabotaged computers in Russia and Belarus, countries involved in the invasion of Ukraine.
   • This malicious version identified developers based on their IP addresses. If the IP address was traced back to Russia or Belarus, the version would delete files and replace them with a heart emoji.
   • To hide this malicious code, the author, Brandon Nozaki Miller, encoded the changes, making it challenging for users to detect the issue by visual inspection.

3. The Fallout

   • Liran Tal, a researcher at Snyk, pointed out that this act represents a significant security risk for any system using the affected npm package if geolocated to Russia or Belarus.
   • Tal highlighted that the node-ipc author manages 40 other libraries, raising concerns about potential malicious activity in those libraries as well.
   • Many in the open-source community criticized the author’s actions, raising questions about trust and the implications of such aggressive actions on the author’s reputation and stake in the developer community.

4. Protestware Emergence

   • The malicious node-ipc update is an example of what’s being termed “protestware.”
   • Other open-source projects have also released updates protesting Russia’s actions in the war.
   • This incident underscores the potential risks when individual developers can significantly impact many applications through open-source contributions.

5. Past Incidents

   • In January, another incident occurred when a developer’s update to two JavaScript libraries, with over 22 million downloads, caused over 21,000 dependent applications to malfunction.

6. Resolution

   • After the discovery of the malicious code, the developer released updates to remove it from node-ipc versions 10.1.1 and 10.1.2.
   • Snyk advises developers to cease using the compromised package or use an npm package manager to override the affected versions.

7. Snyk’s Statement

   • While Snyk supports Ukraine, they emphasized that such intentional abuse damages the global open-source community, leading them to flag the affected node-ipc versions as security vulnerabilities.

Secure resource and Managed Secrets and Key Management Service

How to store keys in KeyVault

How to get keys in pipeline

Using service principals

• When you open your door to your house, you use a key which is a proxy for authenticating you to access the house (through authorization.) When using an API, you can use an API key, which is similar. It authenticates you to the service, and the API's backend authorizes you to make requests. The issue with this approach is that it is not tied to an identity, that is, whoever has the key is the person who is allowed to make the requests. This is not good because many people can use the same key, or someone can steal it. They are also difficult to rotate, as doing so requires scanning your entire codebase, rotating them, and replacing them.

• Run a static security analysis tool to find secrets currently in the codebase, and then rotate them. Once they are committed then it is too late, it is not possible to erase it because it exists in logs and has violated its RBAC boundary.

• How do I know if a secret has been previously committed to the source code? Use [https://stackoverflow.com/a/48739656/220935]{.underline} to search through the commits. Note that you may want to try other permutations, such as base64 encoding the string (locally) and doing other forms of fuzzy matching. If it has been committed, then it is recommended that you rotate it.

• If you are approving PRs, then it might be useful to have a proof-of-presence (or YubiKey) to show that there is something physically present approving the PRs. This should minimally slow down the development (it takes about 30 seconds or less to authenticate.) Make sure that these work across VMs/screen sharing etc.

• I don't need to know or see what the keys are. A better approach is to dynamically inject them into the environment at runtime, using a secure key provider that stores them in a vault. When requesting them, the provider should authenticate and authorize me to access the keys.

• The issue with this approach however is that the keys are still available to the application, and the application may accidentally log the keys, or a hacker may inject malicious code into the pipeline to access the keys. Additionally, developers may have to copy the keys locally to do testing and may unintentionally leak them. Since there are no ACLs on the keys, anyone with the keys can still use them and waste resources.

• A better approach is to not use keys at all, or, to never allow the user access to see the key.

• What does shifting left on security mean? It means avoiding reactive approaches for security. Security in pipelines usually takes a very reactive approach rather than a proactive approach. For example, secret scanning checks for keys that exist in the source code. I don't think there should even be the possibility of having a key generated by an app for an API. These should be managed identities or identities of some kind that are created in the pipeline's environment through a service connection. There are no tokens or anything to manage, and no secret keys that might have the opportunity to be released to the world. There should, in theory, be nothing for the security token static analysis to report because the tokens do not exist in the source code because there are no tokens to manage and no tokens are provided to the user (e.g., API keys.) Everything is given to the pipeline at runtime with limited lifetimes, and the tokens are revoked once the pipeline finishes.

• The issue with storing API keys in source code is that the keys are no longer subject to RBAC. They can exist wherever the source code exists, which may mean that they unintentionally cross security boundaries.

• If someone compromises your API key, then they can use it anywhere until it's rotated. It is also difficult to determine who has access to it, because requests are just made using the same key, so there isn't a form of identity. Cloud services re-use the same IPs (including pipelines), thus, this makes the challenge of finding out if the key was compromised more difficult.

• You could consider using usernames, but this merely means that a username has to be specified with the key in order to use it. This effectively makes the key longer, but it's trivial to get the username.

• Storing the key in a secure location helps, but the key still exists in plaintext. It might be behind authentication, for example. It's still injected in plaintext and passed to the API, so your program still has to see it. It might be injected into environment variables, which allows attackers to view it. It doesn't matter how secure you store it, whether it's in an HSM, at the end of the day, it's in plaintext.

• A better solution would be to use token-based auth, and authenticate against an identity. It is still possible for attackers to retrieve the token and authenticate against your API. However, you now know precisely when and where the token was compromised, and will know that it expires in a few hours (or less), compared to an API key that might exist anywhere for any length of time.

• Think about what would happen if the API key was released. Would they be able to run up costs? Or, is it restricted to a certain rate limit? Would it be an annoyance, or could they access customer data?

• You'd want to have firewalls in your pipeline to prevent data exfiltration as well. Although this might make it painful to try to set up new dependencies or software that require external resources so it could inhibit development. If the pipeline doesn't need to access external resources, or at least often, then it might make sense to add a firewall with some exceptions. Why can't things be packaged for offline access?

Role-based access control (RBAC)

Permissions

Who to give permissions to

• Auditing permissions

  • Team doesn't need writable tokens if they're just going to download packages (writes should require special perms unless everyone is publishing packages.) Easy to request perhaps.

  • Try to make permission sets for different scenarios. This will avoid giving people arbitrary permissions, and also help understand what people are capable of. It also makes it easy to "undo" a particular permission scenario by just removing that scenario instead of trying to manually identify which permissions were in use for this scenario (as they might overlap.)

Lifecycle management

• Assign permissions to a group or team, rather than to individual employees, to ensure continuity even if someone leaves the company. This might be iffy if the group has a lot of permissions because then anyone can easily get access.

Popular security static analysis tools

• Open-Source Tools

• FindBugs with FindSecBugs Plugin: A static code analysis tool for Java that can identify security vulnerabilities with the FindSecBugs plugin.

• Checkmarx: Although primarily a commercial tool, Checkmarx does offer a limited free version that performs static code analysis for multiple languages.

• Bandit: Focuses on Python codebase and is designed to find common security issues.

• Brakeman: A static analysis tool for Ruby on Rails applications.

• SonarQube: Offers various language plugins and detects many types of vulnerabilities. The Community Edition is free.

• ESLint with Security Plugin: A widely-used linting tool for JavaScript that can also be used for security checks with the right set of plugins.

• Flawfinder: Scans C and C++.

• Cppcheck: Another static analysis tool for C/C++ codebases.

• YASCA (Yet Another Source Code Analyzer): Supports multiple languages including Java, C/C++, and HTML, but focuses primarily on web vulnerabilities.

• Commercial Tools

• Checkmarx: A leading SAST tool that supports multiple programming languages and is designed for enterprise use.

• Veracode: Offers a static analysis service as part of a larger application security suite.

• Fortify Static Code Analyzer: Provided by Micro Focus, it covers multiple languages and offers integration with IDEs and CI/CD tools.

• IBM AppScan: Focuses on identifying vulnerabilities in web and mobile applications, supporting multiple programming languages.

• Kiuwan: Offers a broad range of language support and integrates with various IDEs and CI/CD tools.

• Synopsys Coverity: Supports multiple languages and offers CI/CD integration.

• GitLab Ultimate: Built-in SAST in their Ultimate plan. It supports many languages and is integrated directly into the GitLab CI/CD pipeline.

Commit signing

• I have not had a user-friendly experience with commit signing, but I'm hoping that tools have evolved since then and it is easier to do.

• Is there a way to easily debug common issues with commit signing? Maybe by trying to sign a file manually with your key and verify that it works? Is there an exe that git uses internally to sign the commits?

• [amazon.ca/Yubico-YubiKey-NFC-Authentication-USB/dp/B07HBD71HL]{.underline}

• 

Signing commits using a YubiKey and Git

Signing commits using a YubiKey and Git involves several steps, but it provides an added layer of security to your Git workflow. A YubiKey is a hardware security module (HSM) that can store private keys securely, making it harder for malicious actors to access them. By following the steps below, you’ll be able to sign your Git commits using the private key stored on your YubiKey:

1. Prerequisites:

   • Ensure you have a YubiKey that supports OpenPGP.
   • Install GnuPG (gpg).
   • Install Git.

2. Set Up GnuPG with YubiKey: a. Insert your YubiKey and check if GnuPG recognizes it:

   gpg --card-status
   

   b. If it's a new YubiKey, you might need to generate a new key pair:

   gpg --card-edit
   admin
   generate
   

3. Configure Git to Use GPG: a. Tell Git about GPG:

   git config --global gpg.program $(which gpg)
   

   b. Get your GPG key ID (usually the last 8 characters):

   gpg --list-secret-keys
   

   You should see output that looks like:

   sec rsa4096/1234ABCD1234ABCD 2022-01-01 [SC]
   

   In this case, 1234ABCD1234ABCD is the key ID.

   c. Tell Git to use this GPG key for signing:

   git config --global user.signingkey 1234ABCD1234ABCD
   

4. Automatically Sign All Commits: If you want to sign all commits by default in every Git repository on your computer, run:

   git config --global commit.gpgsign true
   

5. Signing a Commit: If you haven’t enabled automatic signing, you can sign individual commits using the -S flag:

   git commit -S -m "Your commit message"
   

6. Verify Signed Commits: To verify the signed commit:

   git log --show-signature
   

   This will show if the commit was signed and if the signature was valid.

7. Pushing and Working Remotely: When you push your signed commits to platforms like GitHub or GitLab, they should recognize and display the "Verified" badge next to each signed commit, confirming its authenticity.

8. Using Pinentry for Passphrase Prompting: If you’re using the passphrase for your GPG key and working in a GUI environment, you might need pinentry to be set up correctly for passphrase prompting.

   Check your gpg-agent.conf (often located in ~/.gnupg/gpg-agent.conf) and ensure it has the appropriate pinentry program set, for example:

   pinentry-program /usr/bin/pinentry-gtk-2
   

   Restart gpg-agent after making changes:

   gpgconf --kill gpg-agent
   

Remember, while signing commits vouches for the integrity of the commit (i.e., that it has not been tampered with), it does not attest to the quality or safety of the code within the commit. Always review code carefully, regardless of its signature status.

If you do not have a YubiKey, you can use GPG with keys stored securely somewhere on your computer.

Commit Signing with GPG (Without a YubiKey)

Commit signing in Git adds an extra layer of integrity checks to your project by showing that a commit was made by a particular individual and has not been tampered with. This is accomplished using GPG (GNU Privacy Guard) to sign your commits. Here are step-by-step instructions:

Step 1: Install GPG

• Linux: You can install GPG using the package manager for your specific Linux distribution. For Ubuntu/Debian:

  sudo apt-get update
  sudo apt-get install gnupg
  

• macOS: If you have Homebrew installed, you can run:

  brew install gnupg
  

• Windows: Download and install it from the official website.

Step 2: Generate a GPG Key Pair

Open your terminal and enter the following command:

gpg --full-gen-key

You will be asked for the kind of key you want, its size, and the duration the key should be valid. Generally, the default settings are good enough. Finally, you’ll be asked for your user ID (email) and a passphrase.

Step 3: List GPG Keys

Run the following command to list the GPG keys for which you have both a public and private key pair:

gpg --list-secret-keys --keyid-format LONG

Step 4: Add GPG Key to Git Config

From the list of GPG keys, copy the GPG key ID you’d like to use. It’s the part after the / in the sec row. Next, set that GPG key in your Git configuration:

git config --global user.signingkey [your-key-id-here]

Step 5: Enable Automatic Commit Signing

You can configure Git to sign all commits by default for a repository or globally. To enable it for all repos, use:

git config --global commit.gpgsign true

For a single repo, navigate to the repository directory and run:

git config commit.gpgsign true

Step 6: Add GPG Key to GitHub/GitLab/Other

1. To get the GPG public key, use:

   gpg --armor --export [your-key-id-here]
   

2. Copy the GPG key, beginning with -----BEGIN PGP PUBLIC KEY BLOCK----- and ending with -----END PGP PUBLIC KEY BLOCK-----.
3. Add this key to your GitHub/GitLab account.
   • On GitHub, go to Settings → SSH and GPG keys → New GPG key.
   • On GitLab, go to User Settings → GPG Keys → Add Key.

Step 7: Tell Git About Your GPG Key (Optional)

If you are using different key pairs or your machine doesn’t pick the right one, you can set the GPG program and the signing key for each repo or globally.

For each repo, navigate to its directory and run:

git config user.signingkey [your-key-id-here]

Or globally:

git config --global user.signingkey [your-key-id-here]

Step 8: Verify Your Commits

After these steps, your commits should be signed, and you can verify them with:

git log --show-signature

This should show that your commits are GPG signed.

That’s it! You’ve now set up GPG signing for your Git commits. This adds a layer of security to your project, ensuring that your commits are verified as coming from you.

Introduction to Reproducible Builds

What is determinism and nondeterminism?

• Deterministic builds are builds that generate identical build artifacts when given the same inputs in the same environment.

• Non Determinism thus is the opposite of deterministic, which means that the output could be unpredictable.

What causes non deterministic behavior?

• Non-determinism can be caused by several sources, such as filesystem inodes, threading, I/O accesses, different payloads after downloading software, datetimes, modification dates, intentional non-determinism (e.g., guids.) Usually, modification dates, dates in general, and things that are intentionally non-deterministic (such as guids/ids) are the most common sources. To fix non-determinism, you must make something deterministic. Therefore, make the modification dates the same (or don't package the modification date), and make sure that the inputs are the same each time. For example, if your application is creating artifacts to be put in a tar file, and they are of a different order each time, then instead order them by something that is deterministic, such as their filename. This will prevent issues with re-ordering in the archives. This has a very large scope, because you have to understand file formats and go very deep into the build process to make things deterministic. This requires a lot of time and energy, and in some cases you may have to do a lot of debugging, as some scripts may interact with other scripts in different ways that can't be easily Google'd because they are specific to your setup.

• Sources of non-determinism

  • [Real Time Linux Summit - YouTube]{.underline}

    1. memory cache, tlb misses, SMI's (from BIOS)

  • Configuration files: Often changed and can lead to errors if not managed correctly.

  • I/O operations: Frequent in CI/CD processes, potential for errors.

  • File system operations: Common, can lead to issues.

  • Access to external services: Calls to databases, other services are frequent.

  • Networking: Network-related issues are relatively common.

  • Use of environment variables: Often used and can be misconfigured.

  • System time: Time differences can affect synchronization and scheduling.

  • Time-related functions: Used for delays, timeouts, etc.

  • Creating tars/zips, different versions of software: Specific to deployment tasks, less likely to be an issue in builds.

  • OS scheduling policies: Can affect the order of job processing.

  • Memory allocation: Depending on the tasks, could be an issue.

  • Non-deterministic thread scheduling: Multithreading is common but the OS scheduler may not always behave as expected.

  • Race conditions: Likely if improper synchronization in concurrent settings.

  • Context switching: Likely to happen but usually well-handled.

  • Signal handling: Used less frequently, but still possible.

  • CPU cache: Even less likely, specific to certain types of jobs.

• This might be as innocuous as changing the date inside of an executable to the time it was built, or something that changes the actual code itself, such as a simple bit flip. These would normally appear as small changes to the executable code. But bit flips can have dire consequences, especially if they are done maliciously.

• Concurrency is only possible when tasks are interleaved, dictated by the OS scheduler. Given that the OS scheduler is not deterministic, therefore, tasks may not be deterministic.

  • This is different from data races, where outputs might be overwritten or whose order matters.

  • For example, if a program that sums numbers doesn't care which array element was populated first, as long as the final array output contains the right elements. This can speed up processes on multi-core processors, which is why concurrency is normally used.

• An OS isn't meant to be deterministic, in fact, that might slow it down. This is because some operations may take less time due to the nature of the data, or the priority of some other tasks.

• There are other forms of nondeterminism that don't relate specifically to concurrency. When writing files, they could have any inode and aren't guaranteed to be in a specific order. File listing operations don't care what order they are, and so go through the files as-is. Unfortunately, since they are not sorted, this means that the outputs could be different, and therefore the data could be processed differently. Linux has an I/O scheduler that's not super related but batches up changes so that it is more efficient to write them to disk, maybe briefly mention that.

• There are multiple layers of nondeterminism, filesystems, listings of files, threading, cores, network, delays, interruptions, other tasks, scheduling, etc.

• To what extent should builds be reproducible on developers' machines if the CI server is reproducible?

Reproducible & Deterministic Builds

• "A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts" - [Definitions --- reproducible-builds.org]{.underline}

• It works on my computer! Why doesn't it work on yours? Having predictability and consistency is of paramount importance for single developers, and those who work on teams because you have to be able to run your program and get the same output to make sure it is working correctly. The term was not yet coined at this stage, and was rather a function of just developing software and the ability to not be chaotic. The ability to recreate the same outputs is of critical importance, otherwise it is unclear what the output should be, and whether the program is working correctly. If other developers are unable to reproduce the build (i.e., even the ability to generate viable build artifacts), then they can't integrate their changes.

• Reproducible builds go a step further and ensure that the actual binaries/build artifacts are identical on a bit-for-bit level (or a strong hash.)

• Reproducibility can be thought of like a supply chain inside of a supply chain. If you look at the entire software development process, you can model it as a supply chain. For example, there are a set of inputs (e.g., packages), some transformation/development in the middle, and then output(s), such as build artifacts. A CI/CD pipeline is like a mini-supply chain. It takes a set of inputs (e.g., your package), builds it through a few steps, and then produces output(s). Each step in that chain may or may not be reproducible. For example, say that you're compiling some code, it generates some EXEs for example (which are reproducible), and then another step stamps the current timestamp in the file. This would mean that the second step in the process is not reproducible.

• Reproducible builds are deterministic builds, but across different environments that may or may not have the same exact environment, but the dependencies are the same. For example, different locales, timezones, filesystems, etc. but the commands to create the build are identical, and the versions are identical.

• When we start talking about reproducibility, we are going further than merely being able to have developers build and run the client application on their computer and all of the features working as intended. It is about having the underlying build artifacts being identical on the filesystem.

• Reproducibility is a gradient as it is not possible to know for sure if something is truly reproducible forever. For example, the kernel could change, there could be a one in a million race condition, etc. It depends on the effort required to make the build match a hash. If the hash is agreed upon for multiple parties, then it can be accepted as the truth.

• Stronger version of reproducible builds: cryptographic signing

Importance of Reproducible Builds

• You want to have some level of reproducibility, otherwise it might not be clear what software you're shipping or if the testing has the capacity to be done correctly (given it could be done on different versions.)

• Why reproducible builds are important, one bit flip can change app behavior for example

• The goal is reproducibility is to ensure the binaries/build artifacts are identical between different builds, as even one bit could cause a completely different program behavior. This behavior could be malicious. It's possible that a single bit might not do anything, or it could. Another goal is to prevent bugs due to the build software. For example, if it's creating a different binary each time, is there an internal threading issue/bug in the compiler/build script that might be unintentionally changing the program's behavior, thus, it is not possible to instill confidence in it due to the fact that the application is different each time? It is also important when you are building source code from the source, because it shows that the package has not been modified in transit from the developer's machine and you can trust the source code to some extent. If building using multiple different computers and different compilers, it prevents a malicious compiler from inserting code into the compiled program. This is because all compilers would have to be infected with the code, thus, making the attack much more complex. Theseus's ship analogy here on what is considered the same program.

• Why reproducible builds? What problems does it solve?

  • Security. If the build artifacts are different every time, then there might be something injecting something in the source code or binary.

  • Improving collaboration and trust in software supply chain

    1. Shows, to some degree, that a build can be re-created in the future (for example, if there needs to be patches or to reproduce a bug), and also if the compiler is sane (i.e., if it doesn't contain malware, although this is debatable.)

  • To show that the source code can match the binaries. This is important because binaries do not have to correspond to their sources (i.e., if you publish a binary and claim to bundle its sources but it contains malicious software, there is no way of knowing.) The other consumers of your package can't verify it for security flaws as easily.

  • Theseus's ship analogy, to what extent is software the same if the output is different? Two pieces of software can run identically if their binaries are not the same, but adopting Kantian philosophy can be helpful because it allows no room for deviation. This means it is difficult for an attacker to sneak in malicious code.

• increased debuggability

• can bisect the pipeline easier

• security (a hacker can't inject something that wasn't there before)

• reliability (the same build is built on the dev machine as is on the ci)

• Builds do not have to be 100% identical to be functionally identical. However, having them 100% identical means there is no interpretation or possibility that the code has to do with non-determinism.

• Also if customers report bugs in the software, you have to be able to reproduce them locally (with the source code) to rebuild the software product at that point in time

• Reproducibility isn't all about security, it is about re-creating the same application. This is very important because in order to use the application or to test it, you have to have something that is able to be reproduced. Developers will be unable to make sure that their features work as intended if they cannot run the application in the same way as other people on the team, and it will be unclear what version of the application is shipped to customers. This can make diagnosing bugs very difficult, including rollbacks/roll forwards, because a different version of the application is built each time. A bug might not exist in one copy of the application, and it will be impossible to recreate it from scratch because it is not reproducible.

• Reproducible builds go a bit farther than merely having the build artifacts being generated and run. It is still possible for build artifacts to differ, even if the VMs are stateless and contain the same filesystem/input files. For example, build software may use random ids or timestamp files, causing them to be different. Two applications may produce identical output, pass all of the tests, but still might not be reproducible or deterministic. There might be slight differences in their code, that, when run, might produce the same output but it contains different instructions or different metadata.

• Even if the file is on a trusted source and might not be compromised (but see Handbrake as a counterexample), it could have a silent version change (or a DNS takeover) and the file will change. This means that the inputs/process is different, potentially causing the build to be non-reproducible.

/// Start of Selection

media.ccc.de - Reproducible Builds

Wayback Machine (archive.org)

NVD - CVE-2002-0083 (nist.gov)

Case Study: Understanding the Importance of Source and Binary in CI/CD Pipelines

Background

The subject of this case study aims to delve into the intricate relationship between source code and its compiled binary, highlighting the avenues through which bugs and vulnerabilities can be introduced. The study presents real-world examples, one dating back to 2002 in the OpenSSH server, and conducts a demonstration using a kernel mode rootkit.

The OpenSSH Vulnerability

A bug was identified in 2002 in the OpenSSH server. This bug stemmed from a 'fencepost error'—an off-by-one error—where the programmer mistakenly used the "greater than" condition instead of "greater than or equal to." Upon fixing, this seemingly significant vulnerability turned out to be a difference of just a single bit in the binary. The root cause was identified by comparing assembly and compilation outputs of the vulnerable and the fixed versions.

Key Finding:

A single bit can make the difference between having a remotely exploitable bug and not having one. The study emphasizes the importance of getting the conditions right in the source code because even a small error can lead to catastrophic outcomes.

The Kernel Mode Rootkit Demo

To further the argument, a kernel mode rootkit was written to demonstrate the potential risks during the compilation process. The rootkit altered what source code was read during compilation without changing the actual files on the disk. This meant that standard file integrity checks like SHA-1 sum would report the source code as unaltered, while in reality, a compromised version of the code was being compiled.

Key Finding:

This demo emphasized that the trust we put in our compilers and build environments can be misplaced. Even if the source code on disk is correct, malicious actors can intervene during the compilation process to produce compromised binaries.

Implications and Recommendations

1. Critical Code Review: Even a small mistake in the code can lead to severe vulnerabilities; therefore, rigorous code reviews are vital.

2. Binary Analysis: Going beyond source code, a close inspection of the binary could add another layer of security.

3. Integrity Checks: Trusting the build environment is not enough; integrity checks must be more sophisticated and include in-process monitoring.

Conclusion

This case study stresses the need for increased vigilance at both the source and binary levels to minimize the risks of introducing vulnerabilities in a CI/CD pipeline. From small syntactic errors to rootkits affecting the compilation process, the risks are real and varied, and comprehensive security measures are the need of the hour.

Concrete Examples of Non-Determinism in a Sample Program

// example of non-determinism
async Task<Task> DoWork(int id)
{
    await Task.Delay(TimeSpan.FromSeconds(1));
    Console.WriteLine(id);
    return Task.CompletedTask;
}

var tasks = new List<Task>();

for (int i = 0; i < 10; i++)
{
    tasks.Add(DoWork(i));
}

await Task.WhenAll(tasks.ToArray());

// produces different output depending on when you run it

Sleep is only sleeping for a second; however, it is not exactly a second each time. This is probably jitter, which is not really related to non-determinism.

// example of jitter
async Task DoWork(int id)
{
    await Task.Delay(TimeSpan.FromSeconds(Random.Shared.Next(0, 5)));
    Console.WriteLine(id);
    return;
}

for (int i = 0; i < 10; i++)
{
    await DoWork(i);
}

• How do I make my builds deterministic?

  • First, check if your builds are already deterministic. Run your build process on a few different computers running the same build tools and compare the hashes of the outputs. You can use diffoscope for this.

  • Managing build environments: It's important to ensure that the environment in which builds are run is as consistent as possible across different machines. This can involve using virtual environments, containers, or other tools to isolate builds from the host system.

    • If everything is different, which build is considered the standard?

      1. Get everyone's exact build process steps, just collect steps/workflows. You may have to get screenshots in case there is version info or any potential miscommunications

      2. Look at each step and question its purpose and why it is required. If it is not required or does not make sense, flag it for investigation. Look into best practices, depending on your application.

      3. Combine all workflows.

      4. Lock all deps, including transitive dependencies. For example, use a package-lock.json file or yarn.lock for JavaScript projects.

      5. Use a new process, and troubleshoot if anyone is not able to reproduce it correctly.

      6. Store all deps in a centralized read only location

      7. Use reproducible build options for your build tool.

    • Auto-updates: This can be tricky to manage, as it can be difficult to know what version of a dependency is being used, especially if it is being updated automatically.

    • Managing build logs: This can be helpful in troubleshooting issues with reproducibility, by comparing the logs of different builds.

    • Storing and tracking build information: This can be useful for maintaining a historical record of builds, and for troubleshooting issues with reproducibility.

    • Python pipenv package has a pip.lock file or a similarly-named file that allows for repro'able builds. Also nix is popular.

    • Using deterministic packaging tools: Some packaging tools, such as nix or guix, use deterministic build processes that produce the same output for the same input, even across different machines. This can help ensure that builds are reproducible.

    • Take all of those inputs that generate those file(s) or event(s) and then check if those are firing in a deterministic way.

    • Ordering/sorting can help make things deterministic.

    • Be careful not to go too far, only the final outputs themselves need to be deterministic. For example, internal workings of your application, as long as they are read sequentially, don't need to be single threaded. This will slow things down a lot.

    • Double check calls to specific APIs to diagnose issues where calls that are returned do not align. For example, random numbers and date/time. Start logging outputs, and then if those outputs, when re-run, don't align, then log their dependencies, and continue until the source is found. You may want to assign each log an ID, and then sort the logs afterwards. Then, you can determine where the problem lies by cross-checking the logs.

    • It is difficult to eliminate all forms of nondeterminism, and this isn't really related to thread safety, as something could be thread safe but still nondeterministic. Adding elements to a thread-safe collection is thread safe, but reading them back is not.

    • Auto log all return values from all functions? TTD from Microsoft might help if snapshots can be compared.

    • Trying to make things too deterministic (when it is not needed) can slow down the program.

    • To support reproducible and deterministic builds, you have to version everything because the inputs have to be the same. The best way to do this is to have everything pre-installed in a docker image, and then just upgrade it when you want to upgrade a dependency. This does require extra management, however, so there is a tradeoff.

  • How much nondeterminism is needed?

    • Verifying digital signatures only on JAR files [media.ccc.de - Reproducible Builds]{.underline}

    • Software BOMs might help with this

    • Builds (should) according to the talk [media.ccc.de - Reproducible Builds]{.underline} be reproducible irrespective of a kernel (e.g, BSD, Linux, Ubuntu, etc.) because this would mean an attacker would have to make a backdoor in all of those kernels for a change to be unnoticed

  • DSC powershell for Windows could be useful

  • [Deterministic.js]{.underline}

  • Higher level concept of reproducibility is quality, as you can imagine an assembly line creating products, if they are all misshapen then quality control cannot accurately assess the quality, and customers might not be happy. There isn't a threshold or understanding of what the product should be thus it cannot be evaluated against quality standards. Creating a prototype vs. an assembly line?

Tools and Strategies for Debugging and Diagnosis

• Important to know what is and is not relevant to determinism, and which determinism is applicable

  • The rate at which packets arrive when downloading a file doesn't matter, as long as the file is intact. The rate at which the packets arrive may be nondeterministic, but this nondeterminism is irrelevant. However, it may be relevant in another context (i.e., VoIP where the rate of packets matters, but whether they are intact is less important, as long as the receiver can hear it.) Both delays are non-deterministic in those examples, however, the importance of latency is much higher in the latter case. If the packets arrive somewhat nondeterministically in VoIP, then it can be fixed, in TCP, well, they can arrive nondeterministically but then they are re-ordered prior to application delivery. So here there are multiple layers, some which are irrelevant.

  • You have to be aware of all of the layers of nondeterminism, but must use an executive decision-making process to understand which layers are important at which time.

  • Also depends on the extent that two objects are the same, and whether this counts against something being non-deterministic. For example, do the file(s) or resource(s) have to match precisely? The common definition is yes, they must match exactly (so that there is no interpretation for error.)

• Excluding files that are not part of the final artifact

  • For example, generated files or log files which are not essential for the final build (files that can be removed and the app will still function normally)

• Going too far in terms of reproducibility

  • Debugging symbols may require that the assembly has a unique uuid, make sure not to stamp on a fake one just to make the build always reproducible (there are better ways)

  • Make sure application still works after doing changes

  • Understand impact of changes and what other things depend on those metadata items

  • Makefiles for example depend on modification dates (I think) to determine if the build should be re-run. Setting the modification dates to the same might interfere with that.

  • Symbol servers might not be able to identify which assembly belongs to which exe

• When to change the environment vs. changing yourself?

  • If your app can't possibly know what the next step will be, then change the environment. For example, a file compressor nondeterministically adds files to an archive. It would not make sense to make the file compressor detect that it is compressing this app and do something special.

  • If your app requires the environment to change its behavior, then change the app. For example, LC_ALL variable defines the locale. If the app must be in different locales, then the app can't have a fixed LC_ALL variable because it has to be built for different languages.

• Verifying

  • Run the pipeline more often to spot failure trends if something is truly unrepro'able when running just a few times

  • Version everything and use package managers when possible (or hashes of packages/installers/versions)

  • tar -W (verify after archiving) important because corruption could cause irreproducibility

    • disable tar remote files and maybe globs in files because I can rsh into my server by default which is weird

  • Making everyone use the same OS could be better for reproducibility but might not be possible. This means that more effort has to be put into the build system (or vice-versa.)

  • If the builds do look ok, then take more and more samples and compare them

Troubleshooting reproducibility issues

• When aiming for reproducible builds, it's essential to check for consistency at various stages of the build process, not just the end. This approach aids in pinpointing issues if any arise.

• Take, for instance, the compilation of a program. Once compiled, you can cross-check the build artifacts to see if they're consistent across multiple builds. However, other steps like encryption or digital signing might introduce inconsistencies. To handle such scenarios:

  • 1. For Encryption: After encrypting, decrypt the application and compare it with the original. If they match, it's likely reproducible unless there's an issue with the decryption tool.

  • 2. For Signing: Remove the digital signature and then verify the application's consistency.

  • 3. For Obfuscation: Use a consistent seed, preferably derived from the application's state before obfuscation. However, this depends on your security strategy and the capabilities of the library you're using.

• If you receive inconsistent artifacts from a third-party:

  • - Determine the reason. Are they providing updated versions, has their server been compromised, or are they delivering different versions for tracking reasons?

  • - Engage in discussions with the software provider. If they're not cooperative, consider switching to a different supplier.

  • As a workaround, store the build artifacts you obtain and use those for subsequent builds. This ensures that unexpected changes don't occur in between builds.

• Step 0: teams

  1. If you're on a team, then get each team member to build the software 10 times (preferably on different days of the week) and then submit all artifacts to a central repository. Label each one by their name.

  2. Use diffoscope to diagnose whose machines are having issues. For example, check if each developer's build is reproducible, or if they are not reproducible cross-section wise. Put artifacts on a central repository exactly as-is (for example, if delivered as a dmg, tar.gz, zip, etc. even if it is an uncompressed folder.) If each developer can reproduce the build, but together they have different builds, then this could signal different build tool versions or frameworks. If each developer can't build reproducibility on their own machine, then this could be both framework issues, or something with the application build process itself. If there's only a handful of developers who cannot match the team, then it might be because of their environment. Try using containers to standardize the deployment and deps. If there are groups of developers whose builds match, then there might be an environmental issue.

  3. To track down the issue, make sure each developer's frameworks are exactly the same. You will have to get versions of all tools, frameworks, OSes, patches, etc.

  4. You may have to play peekaboo with your archives.

  • If you save the hash of the file after it is downloaded, then you can theoretically find the same file if the file disappears from the internet

  • [Rule of three (statistics) - Wikipedia]{.underline}

• Step 1: attempt to find the non-reproducible layer(s)

  • Rationale: Digitally signing multiple identical artifacts could give different hashes (I think) so look inside of the container when doing the comparison. The reproducibility issues may occur at any layer(s)

  • [onekey-sec/unblob: Extract files from any kind of container formats (github.com)]{.underline}

    1. sudo docker run -it --rm --pull always -v ~/extract:/data/output -v ~/files:/data/input ghcr.io/onekey-sec/unblob:latest /data/input/test.bin

  • If it's a compressed file, decompress it

  • Recursively compare all files, and repeat if needed

  • Disable any code obfuscators or protectors

  • Keep hashes of all files on the filesystem and compare them together if the build changes (thus checking if they are part of the dependencies)

  • Pictures are made of pixels but they can be the same except for one pixel, compare them using image comparison tools (jpegs are more complicated because one pixel can affect others)

  • [Pseudorandom Number Sequence Test Program (fourmilab.ch)]{.underline} for guessing if a file is compressed, encrypted, or both

  • Pick higher-level tools when necessary. For example, comparing two PDFs could mean to convert them to text first, then compare them. Or convert them to an image and compare them.

  • Go to the file layout manual and check if there is overlap with changes in the file with what the diffs show

  • Do a three-way or four-way diff to determine if there is a pattern to the changes

  • If in doubt, try to change things and see if it fixes it or changes other parts of the file

  • 

  • 

  • 

  • 

  • This corresponds to the [tar (computing) - Wikipedia]{.underline} modification date section in the header (plus header checksum) that is different. This means that the file has a different modification date. Look into why that is the case and if tar has any options to disable modification dates.

+----------------------------------------------------------------------------------------------------------------------------------+ | .\fq '.. | select(scalars and in_bytes_range(0x123))' test.tar | | | | |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15|0123456789abcdef012345| | | | | 0x108| 61 6c 65 78 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00| alex.................|.files[0].uname: "alex" | | | | 0x11e|00 00 00 00 00 00 00 00 00 00 00 | +==================================================================================================================================+ +----------------------------------------------------------------------------------------------------------------------------------+

1. fq does have a way to diff binary files, however it says that the tar file I'm using is broken (although it is ok.) I may have to use binwalk to find the offset, then use fq to figure out what is at that offset. [fq/usage.md at master · wader/fq · GitHub]{.underline}

• Step 2: adjusting your focus on comparing a layer

  1. Focusing too low means looking at unknown binaries that have a bunch of meaningless changes

  2. Focusing too far means that there was a change, but it could be anything

  3. Using the "file" command to identify what file it should be (or is)

  4. Try decompiling a binary into IL code or source code, maybe instructions to reduce diff size and see what changed

  5. For example, it could be a GUID or a line of code from a library that caused the entire binary to change

  6. Instead of publishing a single file, try publishing a debug or release build that contains all of the libraries as separate files. This will help narrow down which file is different.

  7. Filesystem metadata is (usually) the cause

  8. Some files might be hiding (e.g., resource forks or invisible files) that are only visible when the file is compressed

  9. If the entire file changed, and it isn't possible to go in it deeper, then try to run it multiple times and see if there is a pattern

  10. Is the file digitally signed or encrypted? This would cause the entire file to change if even one byte changes.

  11. For Docker, [docker: extracting a layer from a image - Stack Overflow]{.underline} extract each layer and then do diffoscope on the tar files (assuming that tar files don't need to be composed first). The issue is that the layers themselves are diffs (so I'd need to diff the cumulative sum)

• Step 3: intra-layer comparison tips

  1. If the files are identical, but only change when zipped, then the archiver is doing something to the files

  2. Try compressing with just store and see if the issue persists (files might be re-ordered.) you can use the hex edit tool to view file contents and see if they are rearranged or if there are any patterns.

• Non-reproducible debugging steps (meta-non-reproducibility)

  1. Filesystem ordering (inodes, etc.) can cause files to be ordered differently

  2. Locale settings

  3. Datetime settings

  4. Last modified dates

  5. Non-deterministic simulations (e.g., Monte carlo), use a fixed seed instead

  6. Adaptive compression algorithms (e.g., zstd --adapt I think)

  7. File system fragmentation (when compressing entire filesystem as a blob)

  8. It's like a forensic metadata search in a way

  9. Appending files to tar files may cause them to become silently overwritten when decompressing, making things weird

  10. More ram (or less) can impact the compressor's dictionary size, which could change how files are compressed (better or worse)

  11. Save as much system info for each container after each build

  12. Middle ground: adding a deterministic flag?

  13. Files that are renamed between builds (e.g., webpack dist bundles) might be very similar or the same. Make sure to compare them.

  14. [https://stackoverflow.com/a/64606251/220935]{.underline} for deterministic Jest tests

  15. strace -f -t -e trace=file ./node_modules/.pnpm/node_modules/.bin/jest --maxConcurrency=1 -w 1 2>&1 | grep -F '/dev/urandom' -B 5 run it on chunks of tests individually and see where randomness is introduced

  16. strace while the debugger is active and then pause program execution when /dev/urandom is hit? hmmmm

  17. Use jest.seed instead of Math.random for testing (so that it can be fixed through CI to a predetermined random value)

  18. [diffoscope: in-depth comparison of files, archives, and directories]{.underline} can show where, but the how is important as well

• Additional troubleshooting tips

  1. Use strace to determine which files are being used

  2. strace -xx -yy -e read,openat -o output.log bash -c 'cd .. && ls'

  3. grep -oP 'openat\([A-Z_]+, \"\K([\\x0-9a-f]+)' strace_output.log

  4. Also need command to programmatically parse read calls (output is in hex) so need to decode it

  5. Code obfuscation and/or encryption can cause the artifacts to appear completely different each time. Make sure that the intermediate stages are reproducible before encryption. However, since the encryption layer can't in and of itself be verified, there is a level of trust required at that stage (e.g., The encryption process inserts untrusted code.) Deterministic obfuscation (e.g., a seed?)

     1. "Code equivalence verifier" but still in research phase

  6. Take hashes of all of those files that are being read and then if they are different then there is potentially issues with library versions

  7. Does it use the network? Those files can change by themselves.

  8. What creates the file? For example, a database might be created by a SQL script and then data is inserted. Could the data be inserted in a different order?

  9. Record hashes of each artifact component individually (in addition to the entire artifact) so that if you lose the artifact, then you will be able to troubleshoot building its constituents if the entire artifact has to be re-built

  10. Take a hash of all files at each step in the process, then run your build multiple times on multiple VMs or computers. You can then horizontally compare the hashes and find out which dependency or file changed, and then determine if that file is relevant to your build process. It will help reduce uncertainty when some of the inputs are different and which step is responsible.

  11. Real-world examples and case studies

  12. [OllyDbg v1.10]{.underline}

  13. [microsoft/Detours: Detours is a software package for monitoring and instrumenting API calls on Windows. It is distributed in source code form. (github.com)]{.underline}

  14. xdelta3?

  15. [API Monitor: Spy on API Calls and COM Interfaces (Freeware 32-bit and 64-bit Versions!) | rohitab.com]{.underline}

  16. [GitHub - codilime/veles: Binary data analysis and visualization tool]{.underline}

      1. This would be useful if you have no idea where the pattern exist in your file, or if there are certain patterns that exist at certain points in the file (e.g., the top)

  17. Take hashes of the intermediate build outputs themselves (e.g., when a program is writing to a file, take the hash of what it is writing and then log the hash.) Then, you can compare the hashes between runs and the ordering of the logging to check if there are threading issues.

  18. xdelta3 shows copied binary segments (e.g., files were concatenated non-deterministically)

+--------------------------------------------------------------------------------------------------------+
| alex@DESKTOP-7M8V9ET:/mnt/c/users/Alex Yorke/Desktop$ xdelta3 printdelta test1-3-2-4_to_3-4-1-2.delta |
| |
| VCDIFF version: 0 |
| |
| VCDIFF header size: 41 |
| |
| VCDIFF header indicator: VCD_APPHEADER |
| |
| VCDIFF secondary compressor: lzma |
| |
| VCDIFF application header: test3-4-1-2.data//test1-3-2-4.data/ |
| |
| XDELTA filename (output): test3-4-1-2.data |
| |
| XDELTA filename (source): test1-3-2-4.data |
| |
| VCDIFF window number: 0 |
| |
| VCDIFF window indicator: VCD_SOURCE VCD_ADLER32 |
| |
| VCDIFF adler32 checksum: 18C50DDD |
| |
| VCDIFF copy window length: 40000 |
| |
| VCDIFF copy window offset: 0 |
| |
| VCDIFF delta encoding length: 31 |
| |
| VCDIFF target window length: 40000 |
| |
| VCDIFF data section length: 0 |
| |
| VCDIFF inst section length: 12 |
| |
| VCDIFF addr section length: 8 |
| |
| Offset Code Type1 Size1 @Addr1 + Type2 Size2 @Addr2 |
| |
| 000000 019 CPY_0 10000 S@10000 |
| |
| 010000 035 CPY_1 10000 S@30000 |
| |
| 020000 019 CPY_0 10000 S@0 |
| |
| 030000 051 CPY_2 10000 S@20000 |
+========================================================================================================+
+--------------------------------------------------------------------------------------------------------+

19. Windows-specific tips

    1. Process Monitor

       1. Tools > File Summary > By Folder, select your build folder, then start clicking on files to see which process modified them. Then, record names of all processes and filter events by those processes to see what libraries they are reading, registry keys, potentially other installed software, missing files, etc.

    2. Clone the entire computer as a VHD and then do build there to see if it is reproducible (this is only if you're out of options or if the build is so complicated that it requires this level of isolation.) Also isolates potential issues with hardware if run on another machine.

20. Environment vars

    1. PATH, LD_LIBRARY, etc

21. tar

    1. --sort flag, -W, and

22. Different architectures and OSes

    1. Difficult problem and doesn't have good solutions yet

• If you cannot ssh into CI pipeline, then copy entire env to a VM and then do the build there (and check why it is failing), might be difficult with workspaces (but you could try calling binaries directly), at least narrows down the issue a bit more (plus more debugging tools because it's not a container)

• For external dependencies, try to periodically archive the offline installer if one exists. If there isn't, or the installer downloads external deps, then do [https://askubuntu.com/a/857845/23272]{.underline} (mksqshfs) on the system after it has installed the deps, compress the filesystem and back it up. This will keep everything exactly as-is (you may also want to run a test script prior to the backup to make sure that the deps are all installed and working.)

• tar depends on ordering, so files with new inodes (e.g., ones sponge'd over top of each other) will be archived in a different order (even though the archives are reported to be the same)

• diffoscope output can be confusing if files are re-ordered

  1. alex@DESKTOP-7M8V9ET:~$ diffoscope --force-details test.tar test2.tar

  2. --- test.tar

  3. +++ test2.tar

  4. ├── file list

  5. │ @@ -1,2 +1,2 @@

  6. │ --rw-r--r-- 0 alex (1000) alex (1000) 2537924 2023-01-11 05:24:43.000000 sample2.data

  7. │ -rw-r--r-- 0 alex (1000) alex (1000) 2537924 2023-01-11 05:09:02.000000 sample.data

  8. │ +-rw-r--r-- 0 alex (1000) alex (1000) 2537924 2023-01-11 05:24:43.000000 sample2.data

  9. strace -D -t -f -y -e open,read,write -P /root/test/4019 tar -cf archive.tar test get call that modified file

  <!-- -->
  

  1. RepTrace can do it on linux [send (ohiolink.edu)]{.underline}

<!-- -->

• What should I do if the software/dep that I use doesn't generate reproducible artifacts (and can't be fixed?)

  • Go back to your original goals for generating reproducible artifacts. For example, if it is for security, do you know for certain that the different artifacts could be infected with a virus? Build artifacts using a dedicated internet-isolated VM if concerned about viruses on a cloud provider, take a hash of the artifact, then download and compare. Use an artifact when building and that will become the trusted artifact to use.

  • If it is for going back in time and being able to rebuild the software, can the artifacts or build process be isolated with lots of documentation on how to build them? Store the artifacts really well with versions.

  • Consider isolating the artifact build procedure from the rest of the build, use hermetic from facebook if possible, contact the vendor for deterministic build support.

  • It is not possible to have a fully deterministic build if there are non-deterministic dependencies. Consider switching tool

• Misc tips

  • If you're still having issues, check build logs (and compare them) to see if there are errors or warnings for example

    1. Docker can't help you forever because it depends on CPU, memory, CPU type, kernel, etc. which are not fully isolated from the host. The host can change and mess up those timings.

    2. lsof for linux or sysinternals for windows to check which process was using a file

    3. Try building it on different versions of an OS. This can help produce more artifacts that could be easier to diff (or find similar changes)

Challenges

• When one tries to prioritize build reproducibility, it's a bit abstract because something that is reproducible does not mean that it is fully reproducible 100% of the time. For example, a threading issue that only appears once every a million builds would be unlikely to appear, even if running the build multiple times. Therefore, you have to assess your risk tolerance and what you are hoping to get out of reproducible builds. To make your builds reproducible, it depends on the project. One KPI to use is to determine how many build artifacts (that you are publishing) are different in-between builds, and how much they are different. This provides a somewhat stochastic indicator on the progress of how to make the build more reproducible. This would be an approximate indicator because the low-hanging fruit to fix the reproducibility issues are likely to be easier, and could fix much of the changes in many of the files. The last bit might be more complex and could require build script changes. Some of the trade-offs are dedicating time and resources into making sure that the build continues to be reproducible. If you want a build that is reproducible from the developers' machines and CI, then they will likely have to have a local copy of CI to build on themselves (e.g., a docker container) and potentially the same OS. This could increase costs and complexity. It also depends on the level of reproducibility required. If the build artifacts change by an-appreciable amount (for example, the modification date is different each time and then this is reflected as a different file after the file is compressed), then one might have a certain sense of confidence that the application is reproducible.

• When we look at the tradeoffs for efficiency and flexibility, one may seek to reduce sources of non-determinism such as threading. This means that if two threads are working together (such as creating files), this means that the files could be created in a different order, thus, causing a chain reaction of issues down the line. A quick approach would be to reduce or eliminate threading to increase reproducibility, which would mean that an application would not receive the advantage of having multiple threads to do work in parallel. (Amdal's law.) It could also make it more difficult to change the build script, because one would have to test it multiple times on multiple different environments to make sure that the build was still reproducible. This would mean that the time to make changes would be slower and developers might be apprehensive that they might break the build's reproducibility.

• Some of the costs when implementing reproducible builds are the act of having to run the build multiple times (to make sure that it is reproducible.) This can cost time and resources. It may have to be run multiple times periodically to make sure that it is reproducible, and some scripts may have to be set up to make sure that the artifacts from multiple builds are the same. To mitigate these costs, make sure that you correctly assess how much risk you're willing to take, and ask your compiler maintainers to support reproducible builds which might help reduce some variability. If you do not need to be ultra confident that your build is reproducible, running it a few times could be useful to gain a somewhat good understanding that the build is reproducible and you don't need to run it hundreds of times.

• Version control is useful for helping with reproducibility because you can go back to a certain point in time and re-create those artifacts.

• The cost of reproducibility

  • Extra maintenance, cost, time, troubleshooting, tooling, version pinning, extra scripts, extra environment changes, docs, training

  • Useful if software is critical, needs to be audited, or has strict security requirements. Or, if you need to test a previous version of some software (rebuild it) to verify a behavior or potential security flaw

Setting goals

• Set metrics, such as the amount of data diff between files that are different and then optimize from there. This might be highly variable, however. You may want to use another number that is more consistent (such as the number of files.)

  • Unreliable build -> Inconsistent -> Non-deterministic -> repeatable -> reproducible -> deterministic -> guaranteed build

  • For metrics, you want to quickly go through your entire program and turn it into stages. From there, you can verify if each stage has been fixed or doesn't appear to have any reproducibility problems. Some of the stages can be interlinked, so might be difficult to estimate work.

Dockerfile reproducibility issues

Commands from your list causing non-reproducibility:

Based on the analysis, these commands from the provided list inherently break reproducibility because they fetch the latest versions of software/scripts, clone default branches, or use volatile base image tags:

1. OS Package Management (Fetching Latest):

• RUN apt-get update && apt-get install -y ... (and variants like apt update, apt install -y, apt-get install -y -f curl, apt-get install -y nginx)
• RUN apt-get upgrade -y (and variants like dist-upgrade)
• RUN yum -y install httpd (and variants like yum install -y ..., yum groupinstall, yum update -y)
• RUN apk add --no-cache python3 \ (and variants like apk add ..., apk -U add ..., apk update, apk upgrade)
• RUN dnf -y upgrade (and variants like dnf install -y ...)
• RUN apt-add-repository ... && apt-get update (Adds external repos, often implies fetching latest)
• RUN zypper ... install ... / RUN zypper ... update ... (SUSE package manager)

2. Fetching External Resources Without Pinning/Checksums:

• RUN curl -O https://bootstrap.pypa.io/get-pip.py \
• RUN wget -O - https://deb.nodesource.com/setup_6.x | bash (and other NodeSource setup scripts)
• RUN curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b /go/bin
• curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash
• ADD https://github.com/simple-evcorr/sec/releases/download/2.9.0/sec-2.9.0.tar.gz /app/sec.tar.gz (ADD with URL)
• RUN curl -sS http://getcomposer.org/installer | php
• RUN wget https://cmake.org/files/v3.18/cmake-3.18.2.tar.gz (Relies on external host stability, though versioned)
• RUN wget https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl (Fetches stable.txt which changes)

3. Language Package Management Without Lock Files/Pinning:

• RUN pip install --upgrade pip && pip install --no-cache-dir -r /requirements/prod.txt (Upgrades pip, and relies on requirements.txt pinning)
• run pip install django && pip install mysqlclient (Installs latest)
• RUN pip install -r /my_application/requirements.txt (Depends on pinning in the file)
• RUN npm install --production --silent (Depends on package-lock.json / yarn.lock usage)
• RUN npm i || : (Same as above)
• RUN composer install --prefer-dist --optimize-autoloader (Depends on composer.lock usage)
• RUN composer global require "hirak/prestissimo" (Installs latest compatible)
• RUN go get github.com/mitchellh/gox \ (Classic go get fetches latest)
• RUN conda-env create environment.yml (Depends on pinning in the file)

4. Cloning Version Control Without Pinning:

• RUN git clone https://github.com/CJowo/pcr-guild-django.git (Clones default branch)
• RUN git clone https://github.com/algargon99/PROYECTO_IAW_GARCIA_GONZALEZ.git
• RUN git clone https://github.com/CORE-UPM/quiz_2019.git
• RUN git clone https://github.com/graphite-project/whisper.git /src/whisper

5. Using Volatile Base Image Tags:

• FROM ubuntu (Implies ubuntu:latest)
• FROM alpine (Implies alpine:latest)
• FROM node:latest (and variants like node:alpine, node:current-slim if they receive updates)
• FROM rabbitmq:latest
• FROM golang:latest
• FROM python:latest
• FROM image (Implies image:latest)
• FROM centos:latest

Are there any more?

Yes, beyond those explicitly in your list, other patterns can cause non-reproducible builds:

1. Time-Based Commands: Any RUN command whose output depends on the time of the build (e.g., RUN date > /build_timestamp.txt, RUN echo "Built on $(date)" > /etc/motd).
2. Randomness: Commands that generate random data during the build process (e.g., generating cryptographic keys directly in a RUN step without a fixed seed).
3. Build Arguments (ARG): If an ARG has a default value that relies on external factors, or if different --build-arg values are provided for different builds.
4. Multi-Stage Builds: If an earlier stage (FROM base AS builder) is non-reproducible, any subsequent stage using COPY --from=builder will also be non-reproducible.
5. Network/DNS Fluctuation: Very rarely, if a command depends on resolving a hostname and that hostname's IP address changes and the command's behavior differs based on the specific IP contacted.
6. Build Cache Issues (Advanced): While BuildKit aims for correctness, complex caching scenarios or bugs could potentially lead to unexpected results, though this is less common than the other factors.
7. Implicit Dependencies: Commands that implicitly rely on the state of the host system's kernel or configuration if that affects the build process within the container (less common with modern Docker).

To achieve reproducible builds, you should always aim to:

• Pin base image versions using specific tags or digests (sha256:...).
• Pin package versions explicitly in apt-get install, yum install, etc.
• Use lock files (package-lock.json, composer.lock, Pipfile.lock, go.mod/go.sum) for language dependencies.
• Verify checksums (sha256sum, etc.) after downloading files with curl/wget.
• Checkout specific commit hashes or tags when using git clone.

Sources

• [On business adoption and use of reproducible builds for open and closed source software (springer.com)]{.underline} sources in introduction are very good

• How does portability fit in with reproducibility?

• [Towards Build Verifiability for Java-based Systems (arxiv.org)]{.underline} page 7 sources of non-determinism

• [framingsbom_20191112.pdf (ntia.gov)]{.underline} general info on sboms

• [Identifying Bugs in Make and JVM-Oriented Builds (arxiv.org)]{.underline} convert BuildFS into pseudo-code that other people can use

• [truecrypt-acsac14.pdf (concordia.ca)]{.underline} "explainable build process"

• [truecrypt-acsac14.pdf (concordia.ca)]{.underline} section 3.3 useful

• [COFF - OSDev Wiki]{.underline} to find out what is inside exe file headers,

• This sort of overlaps with testing (non-determinism) because non-deterministic tests make it difficult to debug whether the application is actually working.

• Differences between Idempotency, impure deterministic, and pure deterministic

• [eellak/build-recorder (github.com)]{.underline}

• [GitHub - VIDA-NYU/reprozip: ReproZip is a tool that simplifies the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in computational science.]{.underline}

• [GitHub - polydawn/repeatr: Repeatr: Reproducible, hermetic Computation. Provision containers from Content-Addressable snapshots; run using familiar containers (e.g. runc); store outputs in Content-Addressable form too! JSON API; connect your own pipelines! (Or, use github.com/polydawn/stellar for pipelines!)]{.underline}

• [GitHub - rerun/rerun: Core rerun. See also]{.underline} [http://github.com/rerun-modules]{.underline} ??

• https://hal.science/hal-03196519/file/SW-2020-12-0293.R1_Zacchiroli.pdf

• 10.1145/3373376.3378519 DetTrace

• 10.1109/ISSREW51248.2020.00044

• [https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1008316&type=printable]{.underline}

• 10.1109/MS.2018.111095025

• [https://www.researchgate.net/profile/Duc-Ly-Vu/publication/359878507_Towards_Understanding_and_Securing_the_OSS_Supply_Chain/links/6254748ecf60536e2354f615/Towards-Understanding-and-Securing-the-OSS-Supply-Chain.pdf]{.underline}

• [https://www.digidow.eu/publications/2020-poell-bachelorthesis/Poell_2020_BachelorThesis_SOAP.pdf]{.underline}

• [https://swc.rwth-aachen.de/docs/bachelor-master-theses/2022-004_BT_Strangfeld/2022-004%20BT%20Strangfeld%20-%20Thesis%20Report.pdf]{.underline} chapter 3

• [https://link.springer.com/content/pdf/10.1007/s10664-022-10117-6.pdf]{.underline} 7.4 mitigation measures

• [https://people.freebsd.org/~emaste/AsiaBSDCon-2017-Reproducible-Builds-FreeBSD.pdf]{.underline}

• 10.1145/3460319.3464797

• [https://link.springer.com/article/10.1007/s11219-022-09607-z]{.underline}

• [DIRENV-STDLIB 1 "2019" direnv "User Manuals" | direnv]{.underline}

• [json-stable-stringify - npm (npmjs.com)]{.underline}

Streamlined Guide to Setting Up a Continuous Deployment Pipeline

This guide provides a concise overview of setting up a continuous deployment (CD) pipeline, focusing on key concepts and best practices:

1. Creating the Pipeline:

• Use your CI/CD tool and connect it to your artifact repository.

• Choose a clear and descriptive pipeline name (e.g., "Production - [Software Name]").

2. Deployment Infrastructure:

• Decide on your hosting environment (cloud providers like AWS, Google Cloud, Azure, or on-premise).

• Key cloud provider offerings include:

  • Container orchestration (Kubernetes services like ECS, AKS, GKE)

  • Serverless platforms (AWS Lambda, Azure Functions, Google Cloud Functions)

  • Infrastructure as Code (IaC) tools (CloudFormation, ARM, Deployment Manager)

  • Monitoring and Logging services

  • Security and Compliance tools

  • Artifact Management services

• Carefully evaluate hosting providers based on:

  • Existing relationships and contracts

  • Support contracts and SLAs

  • Use-case support (e.g., IaC compatibility)

• 3. Continuous Deployment Pipeline Steps:

• Artifact Retrieval: Fetch the correct artifact versions from your repository.

• Containerization (if applicable): Build and package your application within a Docker container.

• Artifact Packaging: Prepare all necessary files, configurations, and dependencies.

• Security Scanning: Scan the built container for vulnerabilities.

• Cleanup: Remove temporary files to ensure a clean deployment package.

• Container Registry: Push the versioned container image to your registry.

• Stakeholder Notification: Inform relevant parties about the deployment.

• Deployment: Automate the deployment process to your chosen infrastructure.

  • Use safe deployment practices like blue-green or rolling deployments for minimal downtime.

• Infrastructure Provisioning: Utilize IaC to manage and automate infrastructure setup.

• Monitoring and Rollback: Implement robust monitoring to detect and address issues during and after deployment. Consider strategies like rollbacks or roll forwards.

4. Release Management:

• Gradual Rollout: Incrementally release new features using techniques like blue-green deployments to minimize risk and impact on users.

• Monitoring and SLAs: Establish comprehensive monitoring to track application health, performance, and user experience. Set and meet Service Level Agreements (SLAs) to ensure application availability.

Key Considerations:

• Feature Flags: Utilize feature flags to control the release of new features independently from deployments.

• Database Migrations: Carefully plan and execute database schema changes, especially in environments with multiple application versions.

• Testing: Perform thorough pre-deployment and post-deployment testing to catch environment-specific issues.

By following these guidelines, you can establish a robust and efficient continuous deployment pipeline that enables faster and more reliable software releases.

What is IaC (infrastructure as code?)

• So, you have your application sitting as build artifacts. That's not super useful to the customer. How do you get it to the customer? Well, it has to be deployed to an environment accessible to the customer, usually via the internet.

• Continuous Deployment (CD) uses the build artifacts from Continuous Integration (CI) and deploys them to production using Infrastructure as Code (IaC). This isn't just about running scripts; CD involves comprehensive processes like testing and monitoring. By leveraging CI artifacts, trust is maintained, ensuring that what was tested is what gets deployed. Essentially, Continuous Deployment spans the journey from a developer's initial feature development to its live presence in production.

• Continuous Delivery, on the other hand, offers the flexibility to release updates whenever desired, without it being compulsory. Regular releases, as advocated by CD, foster resiliency and facilitate rapid adaptation to user feedback. Smaller, frequent changes are easier to manage and rectify if issues arise. Plus, with the unpredictable ways customers might use features, it's advantageous to remain agile and receptive to evolving requirements.

• Note: Reusing CI artifacts in CD instills trust; otherwise, the integrity of the entire CI process would be questioned because the artifacts that were tested are different from what is being deployed.

• When we talk about IaC, it means Infrastructure as Code. It is a way to specify which infrastructure needs to be provisioned for your application to run.

• In the past, this may have been a set of instructions, written down on what the current infrastructure looked like and how to set it up. For example, "Click on this button, type in this field, click on Create VM, name it this, etc.". Documentation quickly goes out of date, and it's error-prone and difficult to follow these steps. Not to mention that any configuration changes, no matter how small, in one environment without updating the docs can cause configuration drift: an unfortunate occurrence for managing complex infrastructure.

• The reason why manual infrastructure deployments are not very CI/CD-like, is because they're complicated. They live in people's heads as a fragmented system. And since computers can't mind-read yet, it's not easily possible to re-create that environment, should something go wrong, or if you want to maintain a known good state. Did we change such-and-such last week? Memory is fickle.

• Why is it related to CD?

  • The CD pipeline would take the template provided in VCS and run the terraform script on your cloud provider and prepare the infrastructure. This should happen all automatically.

• What are the principles or values of IaC?

  • Idempotency: no matter how many times you deploy, you'll get the same thing.

  • Immutable: immutable means something that cannot change. Therefore, instead of updating the infrastructure, which could cause configuration issues, replace everything with a new, deployed copy.

  • Composable. Create other puzzle pieces that fit into other architecture patterns.

• Why should I use IaC?

  • Consistency. Everytime you roll out, it will be exactly the same.

  • Reproducibility.

  • Version controlled, thus, it is a single source of truth. Easily rollback to a previous architecture, find what changed (i.e., auditability), or inspect the current architecture.

  • Speed and a fast feedback loop. Reduce trying to manage your infrastructure and trying to track what you changed manually, which could lead to configuration drift between different environments (e.g., QA/dev and prod.) The issue with configuration drift is that it makes it difficult for developers to have a fast feedback loop, because they can't trust that their changes will work in prod if it works in dev because the environments might be too different. Tracking changes in dev to reflect in prod is also tedious.

  • Security.

• What providers do I have for infrastructure deployments? What are some ways I can run IaC code? There are several tools and providers available for infrastructure deployments:

  • Terraform: A cloud-agnostic tool that uses HCL.

  • AWS CloudFormation: Specific to AWS, it uses JSON or YAML templates.

  • Azure Resource Manager (ARM): Used for deploying resources in Microsoft Azure.

  • Google Cloud Deployment Manager: For Google Cloud Platform (GCP) resources.

  • Ansible: An open-source automation tool that can handle tasks such as configuration management and application deployment.

  • Chef and Puppet: Configuration management tools that allow you to define the state of your systems and then automatically enforce that state.

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Certainly! Infrastructure as Code (IaC) is a practice in which infrastructure (networks, virtual machines, load balancers, and connection topology) is provisioned and managed using code and software development techniques. The "advanced beginner" in software or IT might be familiar with setting up environments manually through user interfaces or direct commands. IaC takes this to the next level, leveraging code to automate these tasks. Here's a more detailed breakdown tailored for an advanced beginner: | | | | 1. Code, Not Manual Configuration: Instead of manually setting up databases, servers, or networks, in IaC, these resources are defined in code files. This is similar to how a software developer writes programs to execute tasks instead of doing them manually. | | | | 2. Version Control: Just like software code, infrastructure code can be versioned. This means you can maintain a history of changes, track alterations, and revert to previous configurations if needed. This is typically managed using version control systems like Git. | | | | 3. Consistency and Reproducibility: By defining infrastructure as code, you ensure consistency across different environments. If you've ever heard the phrase "It works on my machine", IaC helps to solve that problem. Everyone uses the same configuration files to set up their environments, which can significantly reduce discrepancies between development, staging, and production setups. | | | | 4. Automation and Speed: With IaC, tools can read the code files and set up the environment automatically. This can drastically reduce the time to provision or scale infrastructure. No more manual setups or lengthy procedures. | | | | 5. Documentation: The code itself acts as documentation. Instead of keeping separate documentation that details how infrastructure is set up (which can become outdated quickly), the IaC configuration provides an up-to-date representation of the infrastructure setup. | | | | 6. Tools and Platforms: Various tools enable IaC. Some of the popular ones include: | | | | - Terraform: An open-source tool that allows you to define infrastructure in a descriptive manner across various cloud providers. | | | | - AWS CloudFormation: A service from Amazon Web Services that lets you describe AWS resources in JSON or YAML format. | | | | - Ansible, Puppet, Chef: Configuration management tools that can be used to set up and manage the state of servers. | | | | 7. Drift Management: One of the challenges in infrastructure management is "drift", where the actual state of the infrastructure deviates from its expected state. IaC tools can often detect and correct drift, ensuring that the infrastructure remains consistent with the code definition. | | | | 8. Safety and Testing: With IaC, you can apply software testing principles to your infrastructure. Tools allow for validation and testing of infrastructure code before it's applied, reducing potential issues in real-world deployments. | | | | In essence, IaC is the practice of treating infrastructure setup and configuration with the same rigor, precision, and automation as application code. This approach results in more efficient, consistent, and reliable operations, bridging the gap between software development and operations. | +========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================+ +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Instead, we can define what we want our infrastructure to be, in a template, usually called an IaC template. They can be written in multiple programming languages--this one is written in a language called "Bicep".

+-----------------------------------------------------------------------+ | param location string = resourceGroup().location | | | | resource myVM 'Microsoft.Compute/virtualMachines@2019-07-01' = { | | | | name: 'myVM' | | | | location: location | | | | properties: { | | | | hardwareProfile: { | | | | vmSize: 'Standard_DS1_v2' | | | | } | | | | osProfile: { | | | | computerName: 'myVM' | | | | adminUsername: 'adminuser' | | | | adminPassword: 'YourStrongPasswordHere' | | | | } | | | | storageProfile: { | | | | imageReference: { | | | | publisher: 'MicrosoftWindowsServer' | | | | offer: 'WindowsServer' | | | | sku: '2016-Datacenter' | | | | version: 'latest' | | | | } | | | | osDisk: { | | | | createOption: 'FromImage' | | | | } | | | | } | | | | networkProfile: { | | | | networkInterfaces: [ | | | | { | | | | id: resourceId('Microsoft.Network/networkInterfaces', 'myVMNic') | | | | } | | | | ] | | | | } | | | | } | | | | } | +=======================================================================+ +-----------------------------------------------------------------------+

• What you can immediately see here is that there is now a single source of truth: the infrastructure does not live in people's heads, fragmented, it exists, documented, and very clear instructions that computers can understand. If you need a new environment, well, just redeploy. It's squeaky-clean, and brand-new, just like the last one.

• This enables quite a few things.

  • First, we can now deploy our template pretty much anywhere (well, on the same cloud provider, more on that in a bit.) I would like to have a test environment that is essentially identical to production. Bam. Done. Now I can test my code freely, and know that it's likely to work on production.

  • What if I made a mistake, and changed a network adapter but I forgot what settings I changed? IaC templates are part of your repository, and are version controlled with the other code. Therefore, just revert back. All of the changes are logged, so you know what went wrong and how to fix it.

  • All of a sudden, we have more traffic. All we need to do is just deploy the template again (well...)

• Ultimately, they allow for much more control, reproducibility, and experimentation, albeit indirectly.

• This process should be very easy to do. It should be a single button process, with potentially the ability to select which version to deploy. It should be clear which version is the latest version. All versions should be in a state where they can be deployed, because the CI pipeline has validated the changes. Note: depending on your process, you may have to validate the changes through a QA process. Normally, unfinished features are behind feature flags, which allows them to be conditionally enabled only on QA's devices in production. This allows QA to test the features while also not slowing down development on unrelated features, and allows deployment to continue to take place.Let's recap what we have discussed so far.

• The puzzle-pipeline, from inception to art gallery, is much like a software development pipeline. In the puzzle metaphor, the employees were able to send their assistant to check if the picture of the puzzle looks good in the frame in the art gallery. The assistant can work while the employees are still working on the puzzle. In the case of software development, the pipeline provides a set of checks (such as building and testing the code) that provides developers with a baseline level of confidence that their changes work. Instead of sending it to the art gallery via an assistant (who provides feedback quickly), the pipeline is able to run autonomously, quickly and provide feedback. This allows the software developers to make quick changes on the fly.

• Since the pipeline runs after each developer merges their changes, then therefore the application is always integrated, developers have access to other developers' changes, and since the tests ran, it gives a semblance of high code quality and instills confidence in the application.

• Much like the puzzle, the repository is like the puzzle with the grippy board. The grippy board that helps hold the puzzle pieces together while working on it isn't put into the final product, it is just used for development purposes, but everyone who is working on the puzzle needs it.

• This is a collaborative process, provided by CI, that, through code review and small PRs, allow for some semblance of the big picture and allows people to work together to incrementally add the puzzle pieces together.

Continuous Monitoring

[GitHub - microsoft/FeatureManagement-Dotnet: Microsoft.FeatureManagement provides standardized APIs for enabling feature flags within applications. Utilize this library to secure a consistent experience when developing applications that use patterns such as beta access, rollout, dark deployments, and more.]{.underline}

Look at Grafana "exemplars" and "correlations". and application topology map.

• In the world of software development, the role of continuous monitoring can't be overstated. It's the heartbeat that tells us how our applications are performing, where they're faltering, and how users are interacting with them.

• Imagine this: You've just released a new feature. Is it working as expected? Is it meeting user needs? These questions underscore the necessity of a robust monitoring system. But, while having a myriad of dashboards might seem helpful, it's not just about accumulating data. It's about distilling this data into actionable insights, helping teams swiftly locate and address issues.

Why Monitor?

• The purpose of monitoring extends beyond troubleshooting. It offers insights into user behavior, providing key business metrics like daily or monthly user engagement. Such data isn't just numbers; it's a reflection of user satisfaction and product viability.

Characteristics of Effective Monitoring

• Coverage: Traceability through the system is crucial. This means tracking a user request from initiation to conclusion. Correlation IDs or trace IDs can be invaluable in this regard.

• Relevant Data: Log entries should provide meaningful information. Whether it's an error message, user ID, application version, or the server it's running on, every bit of data aids in piecing together the bigger picture.

• Strategic Logging Points: Position logs where they can offer the most diagnostic value.

• Priority Management: Assign importance to your logs, ensuring critical logs don't get buried under the noise.

• Freshness: Updated, real-time data often carries more value than stale information.

Making Sense of Data

• Collecting data is only the initial step. The challenge lies in understanding this data, and will likely take 95% or more of your time. Visualizing it, plotting graphs, and discerning patterns will likely consume a significant portion of your time. Graphs, while they should be comprehensive, must remain straightforward, avoiding needless complexities that could mislead or confuse.

The Importance of Feedback

• Consider a jigsaw puzzle shipped to a customer. How do we know if it reached in perfect condition? Did the colors appeal to the user? Did they find the image appealing? It's this feedback, collected via monitoring, that guides further iterations of the product. Continuous monitoring, embedded within the CI/CD pipeline, offers constant feedback on performance, errors, and user engagement.

Telemetry: A Close Look

• Telemetry is the backbone of continuous monitoring. It involves collecting data from the platforms where applications are run. This data is gathered on both the server, providing metrics like CPU usage and memory consumption, and within the application, capturing user-specific metrics. These might include engagement levels or user satisfaction metrics.

Monitoring Frequency

• By definition, continuous monitoring is unceasing. Data, much like a river, keeps flowing into the central hub, offering a live pulse of the application.

• So the reason why you don't monitor absolutely everything is because there's a cost of monitoring. Otherwise it's just more every single instruction in every single line of code. And the reason is because of that is monitoring is designed to.Be a little bit pragmatic. So you have to kind of know like, OK, what am I trying to actually solve for this?But it was my goal. Am I trying to reason about the program's execution?To find a bug. So for example, am I trying to reduce entropy?With the program execution, when someone runs something, then these logs generated and I can retrace the program steps, which case you don't need to log every single line of code likely.You see the log you know inside of the if statements and potentially some.Variables are useful.Information.Like that and.Yeah, there definitely is a way to avoid logging, which is, you know, just debugging. But.Debugging is kind of more of a.To all that's.Used to kind of fix something as a means to an end and it is kind of difficult to use a sustainably because.Logs provide more context.And there's sometimes that you can't use it to debugger, like you know, if some customer reproduced at some point or something like that layer code or that's never be totally different in the logs can capture that. But you can't go back in time with the debugger and try to figure out what exactly happened. It's very difficult to take dumps and such.Um.So.And another thing is if you matter too much, um, you have to do something with this data. And if you just have way too many logs while Southern application performance, if it's got too much, if you're using like mobile applications, for example, well, you know, it's, you know, you're pumping out like hundreds of megabytes of logs.Over the user cellular connection or the ASB battering ram and such.The other part is actually like how to process it. And if you're logging like way too much stuff, you're spending a lot of these CPU cycles and.Storage and self storing all this stuff.And it's even more difficult to audit as well.Have you said that like you, too little is also?Definitely probably a larger problem so.Log 2 matches. Probably better to do that instead of logging too little I'd say.

• And another thing, sometimes people differentiate it.It's metrics and locking and analytics so.Logging is just kind of the act of just like saying ohh you know, the program breached.This point or something like that, uh, metrics are kind of more about.Ah.Logging things that can be graphed, so for example like CPU usage would be considered a metric. You could have a graph that shows like over time.How much memory using, how much CPU you're using inside, etc. Technically these are logs, but there could be a bit differently to kind of process differently as well. They're not really associated to a code path per se, it's just like.Diagnostics for the for the whole machine.

• And let's try to do this for another application. The first step is with logging. It's normally to help to reduce the entropy of your program state.So in this case.We have the application initializers and displays some stuff to the user. Let's just kind of go over like a very basic logging exercise. Let's also go into correlation IDs to show how you can trace the request back from users web browser all the way throughout the application. What request the application?Makes, et cetera, and we'll see why later, why this is really helpful and important.And this usually requires doing some manual setup with the conference doctor. No per se.Like.That this is associated to this request or something like that.So yeah, I think it's going to be really useful.

• So the first thing I probably do is bring up our code that we have and then figure out where it probably be good spots to do.Some logging and then as we make our application more complicated, we'll see kind of how this scales.Umm.As well and a lot of don't necessarily have committed through every single application this call like you can admit it only like 10% of the time or 5% of the time. Just get like a good understanding of what's happening, especially there's a lot of users hitting that same point. You don't necessarily need like 100% of the time, it's always.Logging that because it could be bad for performance. But again, it kind of depends on your use case. You're probably going through this a little bit more as well.

+-----------------------------------------------------------------------------------------------------------------+ | import React, { useState, useEffect } from 'react'; | | | | import axios from 'axios'; | | | | // Data fetching logic extracted to a custom hook | | | | function useWeather(apiKey) { | | | | const [weather, setWeather] = useState(null); | | | | const [loading, setLoading] = useState(true); | | | | const [error, setError] = useState(null); | | | | useEffect(() => { | | | | async function fetchWeather() { | | | | try { | | | | const response = await axios.get(http://api.openweathermap.org/data/2.5/weather?q=London&appid=\${apiKey}); | | | | setWeather(response.data); | | | | setLoading(false); | | | | } catch (error) { | | | | setError(error); | | | | setLoading(false); | | | | } | | | | } | | | | fetchWeather(); | | | | }, [apiKey]); | | | | return { weather, loading, error }; | | | | } | | | | function Weather() { | | | | const apiKey = process.env.REACT_APP_WEATHER_API_KEY; | | | | const { weather, loading, error } = useWeather(apiKey); | | | | if (loading) return <p>Loading weather...</p>; | | | | if (error) return <p>Error fetching weather</p>; | | | | return ( | | | | <div> | | | | <h1>{weather.name}</h1> | | | | <p>Temperature: {weather.main.temp}°C</p> | | | | <p>Condition: {weather.weather[0].description}</p> | | | | </div> | | | | ); | | | | } | | | | export default Weather; | +=================================================================================================================+ +-----------------------------------------------------------------------------------------------------------------+

Let's explore where it might be beneficial to implement logging in an application. Proactive logging is crucial as it allows for quicker bug resolution without needing additional log deployments. For example, in an application that handles weather data, important log points could include:

1. Initial API Key Check: Verify if the API key is set but avoid logging sensitive information.

2. Conditional Statements: Log within conditions handling loading errors or operational states to trace the application flow and identify issues.

3. Performance Metrics: Log the duration it takes to load the weather data, potentially using Web Vitals to capture timing from the initial request to the display on the user's screen.

4. Error Handling: Implement an error boundary to log errors without crashing the application, providing a fallback UI with support links for a better user experience.

5. Telemetry and Metrics: Beyond basic logging, collect telemetry on user interactions, such as location queries, to inform higher-level management reports and monitor system performance.

Additionally, consider logging retry attempts in server communications to correlate them with session IDs, enhancing error analysis and improving the overall reliability of data capture in your application. This approach to logging not only aids in immediate troubleshooting but also enhances long-term application stability and user satisfaction.

Web Vitals primarily focuses on assessing the performance of applications, particularly useful for single-page applications, though adaptable for others. It measures high-level performance metrics like initial load time and various user interaction metrics to detect performance regressions. Installation and usage are straightforward: simply install the Web Vitals package and integrate it into your application.

It's designed to capture events such as input delays during usage, continuously updating to reflect the maximum input delay observed. This requires careful database query structuring to ensure only the latest event per session is considered, avoiding duplicates. This is because Web Vitals may send multiple events as it updates the maximum observed values while the application is in use. If the user exits prematurely, some data may not be transmitted, although web beacons could mitigate this issue, albeit with limited library support.

Reliability and Performance Metrics

• It's impractical to keep a human eye on the influx of data at all times. This is where automated alerts come in, signaling when intervention is necessary. Using reliability metrics like ICE (Ideal Customer Experience) and ACE (Adjusted Customer Experience), teams can gauge application performance against established benchmarks.

Introduction

• Let's set the scene. You've released a new feature, or want to ensure that your website or app is still usable by the customers. You can use monitoring to make sure that your customers expectations (with regard to automated tests and performance) remain valid.

• There is one thing about monitoring, however. It's likely that your dashboards aren't going to tell you precisely where the problem is, therefore, you should make your code flexible, and develop a good monitoring strategy to know where to log or to debug next. If that was the case, then, well, you better get coding, because you're going to need a lot of dashboards. This might not be a worthwhile strategy. Part of monitoring is about reducing execution entropy and to reduce disorder by tracing execution. It's important to be able to know how to read a dashboard, which metrics are important, which are less important, and how this corresponds with the actual system, including how to trace requests and look up logs.

• Monitoring isn't all about trying to find bugs. It's also useful for understanding user behavior, for example, how many users use the app per day/month or how many users use the platform. These are very important business KPIs.

• Things that a good monitoring system has:

  • Coverage. The ability to trace a request through the entire system. This doesn't mean that you will necessarily know precisely what happens at each step, only that it goes through a system, but it got messed up for example. There has to be a complete path from the user's web browser all the way to the request being serviced, and back again. Teams should provide a correlation id with requests, or provide the capability for you to add your own trace id to the request so that you can track it and helps the other team know if you need help. This might mean that you need to add monitoring to many other intermediary services.

  • Useful data. The events/metrics/logs have to be useful and contain relevant information which can be used to debug. For example, if a user is trying to sign up, but fails, then it might be useful to log their user id or the associated error message. One of the goals should be to reduce execution entropy. Think about it from the person using the logs to ascertain previous system behavior. Are they able to find out where the request failed within your application? How much are they able to narrow it down? It might also include the application's version, along with other versioning information, such as what server it is running on.

  • Useful logging points. This is similar to useful data, but the logging should be in places where it matters, and has a capability to help debug the application, for example, usually before and after control-flow statements, but this depends.

  • Priority/importance. Not all logs are useful, but some are. This doesn't mean you shouldn't log anything that is not critical, it just means to assign a priority to your logs. This allows you to easily filter for the high-important items, providing a better signal to noise ratio.

  • Frequency. Stale or old data is normally less useful than fresh, up to date data, much like a stream.

• Collecting data is the "easy" part. What do I do with all of this data? This is called sensemaking, literally, making sense of the data. IThe act of aggregating, graphing, plotting, and transforming your data is likely to take 90% or more of your time. It's important that you have clear graphs that represent your data accurately, and you might find it useful to graph the same data using multiple data visualization formats multiple times to get different perspectives. [Show Me the Numbers: Designing Tables and Graphs to Enlighten: Few, Stephen: 9780970601971: Books - Amazon.ca]{.underline} This book is intended for how to design boring graphs. Boring graphs aren't a bad thing, you don't want to be distracted by unnecessary visuals which might alter your perception of the data, or to distract you. Graphs should be used to enlighten, not confuse (at least within the technical realm.)

Why is monitoring important?

• After the puzzles have been shipped to our customers, how do we know if they liked them? Were they satisfied? We can put our phone number in the box so that they can call us if they liked it or didn't like it.

• Some of the other questions we'd like answered are:

  • Was the puzzle squished in shipping?

  • Do the colors look nice?

  • Did the image look ok?

• Instead of spending more and more and more energy making the process perfect, which would significantly hinder the integration time (i.e., diminishing returns), we instead try to be resilient and understand how to fix things as they come up, and limit the amount of damage. We expect that there are, at some point, going to be issues. Therefore, we proactively make sure that we have the capability to know when these errors will occur, and limit the amount of customers that are impacted by doing incremental rollouts. We also want to have the ability to know if our customers are using the feature as well, which is important for the business (which would be considered a non-error metric.)

• Continuous Monitoring corresponds to this feedback mechanism: getting feedback. In this case, continuous monitoring refers to getting feedback from a customer's device where they are running our application, in terms of performance, errors, engagement, and more. Developers should embed telemetry inside of their features to ensure that customers are using them, and to quickly turn off the feature flag should there be an error. This is because features with errors could corrupt data, and are not beneficial to the customer's experience. Feature flags are a huge part of continuous integration and CD: they enable developers to quickly experiment and integrate code, all the way to production. Much like a river, events are continuously generated, and continuously logged, and continuously monitored.

• With continuous monitoring, developers will add monitoring to the features that they release, including monitoring metrics for the physical servers that they are deployed to.

Terms

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 5. KPIs: Key Performance Indicators. Metrics that demonstrate how effectively a company is achieving its business objectives. | | | | 7. Correlation id: A unique identifier value that is attached to requests and messages that allow tracing of the whole chain of events of a transaction. | | | | 8. Telemetry: An automated process where measurements and data are collected at remote points and transmitted to receiving equipment for monitoring. | | | | 10. MAU: Monthly Active Users. The number of users who interact with a product within a 30-day window. | | | | 11. SLA: Service Level Agreement. A contract between a service provider and the end-user that specifies what level of service is expected during the agreement's duration. | | | | 12. ICE (Ideal Customer Experience): A metric that measures user satisfaction, calculated as the number of successes divided by starts. | | | | 13. ACE (Adjusted Customer Experience): A metric that considers both successes and expected failures, divided by starts. | | | | 14. Error Budget: An engineering concept based on the premise that 100% uptime or reliability is neither realistic nor the goal; instead, a certain "budget" of allowable errors or downtime is set. | +=================================================================================================================================================================================================================+ +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

What is telemetry?

• Telemetry is a method to collect and send data from customer devices where our applications run. Developers embed telemetry within features to monitor performance, errors, engagement, and other metrics.

• The way continuous monitoring is integrated into your application is two-fold:

  • On the servers hosting the application, there are several integrations or applications you can use to monitor server-specific metrics, such as the number of requests served per second, CPU usage, memory, disk, etc. These are usually agents or applications that run directly on your server, or, it could be provided by your cloud-hosting provider. Normally, this doesn't require modifying the application, but it depends on what type of metrics you want to collect.

  • On the applications themselves, normally, you'd want to collect application-specific or user-specific telemetry. This does require modifying the application to add code to log the specific metrics that you are interested in. There are some frameworks on how to collect this telemetry, and, depending on which telemetry you are collecting, might be sent to a third-party server or your server. If collecting it yourself, it is normally via an HTTP endpoint which stores events into a database that can be queried later (e.g., for reporting.) Some useful metrics that you might be interested in are how many users are using your application per month (MAU), per week, or per day, and whether the user's are happy.

  • There might be several points in-between, such as those from different teams, that may also benefit from monitoring.

Frequency of monitoring

• Well, continuous monitoring monitors continuously. This means that as event data is generated, it is streamed (like a river) to our centralized analytics hub. In some cases, it might be aggregated or batched up on the client's side, but this is an advanced topic.

Benchmarks for reliability and performance

• Of course, someone isn't staring at the data all day, that would not be a very good use of their time. Instead, we can set up alerts or rules that trigger an event (such as a phone call, email, etc. based on the severity of the alert) for an intervention to occur. An intervention is an act that a human does in order to fix or silence the alert; the computer says that something is wrong, and the human has to either fix it or silence the alert by evaluating the situation. Sometimes, you can also set up automatic interventions where a pre-programmed action takes place if there is a certain type of alert.

  • Let's say we have some telemetry for our new feature set up. Whenever there is an error, the error handler triggers and sends us some error telemetry. If there's a lot of people using our application, there's bound to be maybe one or two false positives. Say that there are millions of people using our application. We might not want to wake up every time someone encounters an error, otherwise I would not get any sleep.

  • In the industry, we measure application reliability and success through something called ACE and ICE [Delve Telemetry & Monitoring. My name is Luka Bozic and I'm a... | by Delve Engineering | Medium]{.underline}.

  • "ICE (Ideal Customer Experience) = successes / starts". In this case, we have 999999 successes and 10000000 total starts (one error.) So, our ICE would be 0.999999.

  • "ACE (Adjusted Customer Experience) = (successes + expected failures) / starts". Expected failures are errors that are retried (succeeded), or errors that are technically not "errors". In this case, our ACE would be the same as our ICE.

• What would my ICE and ACE be? It depends on your application, but usually 99.95% is a good start. This really underscores the importance of good monitoring and also bridges the gap between what the customers see, and what is being evaluated against. Monitoring is only as good as what you put into it.

• But, that doesn't allow for much experimentation, does it? Correct. This allows for about four hours and 20 minutes of downtime, per year [SLA & Uptime calculator: How much downtime corresponds to 99.95 % uptime]{.underline}. Going up to 99.99% is about 52 minutes of downtime per year. Note that this normally means that the entire application is unavailable; if items are feature flighted then it is likely that an individual customer(s) will have downtime. Therefore, if you are going to make an SLA, then know that it can restrict how much experimentation takes place.

• Wow, we should be super on the safe side, right? Well, technically. You can take calculated risks, such as by using an error budget which allows the team to perform more risky changes when you still have SLA remaining. This allows customers to expect a certain level of stability, while also ensuring that the team can continue to experiment and deliver features on time. This also helps keep stakeholders informed as to the extent that customers are impacted.

Getting started with monitoring

• It's likely that your application is complicated. Where do we start to collect data? When we think about what we need to collect, we need to start with a user-focused mindset. This normally involves collecting telemetry on the user's side, such as performance, errors, and frequency metrics (e.g., how often a button was pressed.) It's important to think about the big picture about what you're trying to achieve first, and then do the concrete implementation of the telemetry later. For example, say I want to know if the "Create project" feature that is being rolled out meets customers expectations. We know for sure that it can't meet customers' expectations if it doesn't work. Therefore, we can add an error handler to send error data back to our analytics hub should there be issues. We can then set up alerting, or rules on the data, that will tell us immediately if customers' expectations are not being met. This helps with experimentation as you get a very fast feedback loop: as soon as there is an issue, you will be notified usually in the order of a few minutes or less, and can correlate it with what you're doing.

Ok, where do I start?

• First, you have to think about what you're trying to monitor, especially if it is a business case. For example, the business wants to know how much people like the application. This could be broken down into several sub-goals, such as user retention, logins, activity, etc. and then these can be monitored individually, by turning them into scenarios. Identify these scenarios in your app, and then apply logging to those locations.

• Another situation which overlaps is determine if there are issues or problems in your application, for example errors or performance issues. What are the core user scenarios, for example, when they click on your app, how long does it take to load for the first impression? What about some other processes, like creating a project? Does that take 10 minutes when it should take 10 seconds? What is the entire flow from when a user enters the app to that point? This might require logging at many different points, but there should be a well-reasoned strategy, such as logging in places that reduce execution entropy. For example, logging twice in a row is, in general, probably not as useful to determine what happened next than if it was logged after an if statement. Here's an example.

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Certainly! Let's consider an example involving a simple function to process user registration in a hypothetical application. | | | | ### Poor Logging Strategy | | | | python | | | | def register_user(username, password, email): | | | | try: | | | | # ... some code to add user to database ... | | | | db.add_user(username, password, email) | | | | # Vague and non-descriptive log | | | | print("Operation completed.") | | | | except Exception as e: | | | | # Logging the whole exception without context | | | | print(e) | | | | # Usage | | | | register_user("alice", "password123", "alice@example.com") | | | | | | | | Issues with the above code: | | | | 1. Using print instead of a proper logging library. | | | | 2. The success message "Operation completed." is vague. We don't know what operation was completed. | | | | 3. Catching all exceptions and just printing them out without context can make it hard to understand the root cause. | | | | 4. Sensitive information, like a password, might get logged inadvertently in the exception message. | | | | ### Good Logging Strategy | | | | Using Python's logging module: | | | | python | | | | import logging | | | | logging.basicConfig(level=logging.INFO) | | | | logger = logging.getLogger(__name__) | | | | def register_user(username, password, email): | | | | try: | | | | # ... some code to add user to database ... | | | | db.add_user(username, password, email) | | | | # Descriptive log message with relevant context | | | | logger.info(f"User registration successful for username: {username}, email: {email}") | | | | except Exception as e: | | | | # Logging error with context and without exposing sensitive information | | | | logger.error(f"Failed to register user with username: {username}, email: {email}. Error: {type(e).__name__}") | | | | # Usage | | | | register_user("alice", "password123", "alice@example.com") | | | | | | | | Improvements in the above code: | | | | 1. Using the logging module which provides more functionality and flexibility compared to simple print statements. | | | | 2. The success log is descriptive, providing context about which operation was successful. | | | | 3. The error log provides enough information to understand what went wrong without dumping the whole exception, and without exposing any sensitive information like passwords. | | | | 4. It's easy to change the logging level, format, and destination (e.g., file, console, external system) with the logging module. | | | | In practice, a good logging strategy would also involve considerations like log rotation, centralized logging for distributed systems, monitoring of logs for anomalies, etc. | +=================================================================================================================================================================================+ +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

• The reason why you want to include other information in the logs (e.g., successful, and not necessarily just a logging guid) is because you want to be able to quickly glance at the logs in your logging viewer (whatever that is) and be able to quickly discern if there are issues. Otherwise, you'd have to look up the logging ids every time.

• Another important thing is: if there was an error, what was the user's experience? This is more of a software-related question and not necessarily a CI/CD one.

How do we collect this data?

• This would depend on the application that you're writing, but normally there is an endpoint, called /analytics for example that captures event payloads. The event payloads typically correspond to a particular user (e.g., by sending the user's id with the request, along with their session id, time, and what they were doing.) Be careful to read all privacy laws applicable in your area (e.g., GDPR) as some information may have different retention policies or the ability to capture certain types of information.

• Here's a sample code impl in a TypeScript application.

+--------------------------------------------------------------------------------------+ | const telemetry = new TelemetrySender('https://telemetry.endpoint.com/sendEvent'); | | | | const event: TelemetryEvent = { | | | | eventId: 'userClickedButton', | | | | timestamp: Date.now(), | | | | data: { | | | | buttonId: 'saveBtn', | | | | userId: '12345' | | | | } | | | | }; | | | | telemetry.send(event); | +======================================================================================+ +--------------------------------------------------------------------------------------+

• In this case, an event is sent to the analytics server once the user clicks on the button. This event is usually associated with the currently logged in user, particular feature flags enabled in the application, or might be part of the payload itself. This would depend on your specific telemetry's implementation, but should contain enough data to trace the request back throughout the application (e.g., via a correlation id) so that you can debug it.

• This event can be anything: it could be an error, a successful event, or diagnostics (e.g., something neutral.) It's up to you to decide what you should monitor, but focus on what the user sees, and what telemetry might be triggered if a user performs a specific action.

• There are other monitoring tools that are much more high-level. For example, they might try to load your website in a virtual browser, or boot your application in a VM and verify that the content of the website looks the same. If, for example, it doesn't load, then it can notify you. The advantage of also using this layer of monitoring is that if the website does not load, then it is not possible for the client-side telemetry to emit useful data. Or, for example, say the telemetry was successfully emitted, but there was a CSS issue that caused a div to take up the entire screen, making it impossible for people to navigate the website. By collecting data, you are able to notice trends and patterns, and so if there is all of a sudden a lack of telemetry, or much more, then you are able to have historical stats to back it up and then be notified to do an investigation.

How do I process this data?

• There are many tools available to do so. For example, databases that support KQL (Kusto), MySQL, NoSQL, ClickHouse, etc. The act of creating queries is outside the scope of this book, but is likely to take the majority of your time.

What should I monitor?

• Developing a monitoring strategy is important because otherwise the monitoring might not reflect the true user's experience. This can make it difficult to get a fast feedback loop, and for experimentation, as you can't trust your data or dashboard to reliably notify you if there is an error. This means that things like feature rollouts via feature flags, incremental deployments, and more would not be as trustworthy.

• Say a user is creating a new project. Some items that you might want to log or monitor are if creating the project is successful (i.e., did the user just see a white screen? Was there a crash?), how long it took to create the project, what the user was doing beforehand, etc. The errors are usually logged in an error handler, but would depend on the framework that you are logging in.

• There are other levels of where stats should be collected at. For example, the HTTP request itself, such as its status code, latency, etc. This is usually done server-side, and because of the homogeneity of many back-ends, many alerting templates likely will automatically monitor this as a default. These are mostly diagnostic data: for example, 100ms for an HTTP request doesn't mean much in and of itself, or 10% CPU usage, and then fluctuates to 5% for example doesn't mean much either. This is useful for example if you are having issues on the client, or people are experiencing issues and you find out that the CPU usage is at 110%, then it's likely there's a CPU usage issue.

  • However, some are useful for keeping track of. If the CPU usage is steadily rising, with more and more traffic, then you might need to consider your scaling strategy or to provision more servers for example. This provides an early warning sign before issues occur.

• It's also important to collect actual user feedback as well, for example, through a feedback form. This is helpful for collecting higher-level errors, such as usability or new features, which would be difficult to capture via diagnostic data.

• Now you have a whole bunch of events in your database. What do you do with them? They are not useful if they just sit in the database.

• In the case of errors, you'd typically want to create an alert on this. There are several options available, such as DataDog that can help with this. They have different integrations, such as being able to call your phone if there is a certain amount of errors that occur within a timespan. Note that it's only good as your monitoring setup: if you don't do any monitoring, then you won't get any alerts. This doesn't mean that your application is healthy, however.

Importance of goals

• It's likely that you will be overwhelmed with numbers, dashboards, and data. Do I care if one HTTP request took 1000ms, and other one took 1001ms?

There's lots of places to monitor. Where do we start? Well, let's create a strategy.

• Webapps are very complex. Only measuring the HTTP calls is a poor representation of the user's experience, because many HTTP calls can comprise a user's request. Therefore, even a single call per user could lead to a bad experience (or a slow script) which might not be reflected in the time that the HTTP calls are made. Browsers pipeline requests, and can do requests in parallel, thus making it very challenging to reconstruct it.

• web-vitals useful for measuring web-app perf [GoogleChrome/web-vitals: Essential metrics for a healthy site. (github.com)]{.underline}

• Know the limitations of your monitoring tool. If it can only measure status codes, then therefore it might be too granular to use for specific user-experience metrics, such as click time.

• Therefore, it depends on the context and application (and where it is running.) You might find it helpful to try simulating the environment that it might run on, including artificially slowing down the environment to make sure the telemetry is accurate.

• Sometimes, the request can't be served at all. In that case, server side monitoring allows for knowing if there are issues server-side.

• Differentiate between diagnostics and goals. Diagnostics are like checking your heart rate, it doesn't really do anything on its own. Or checking how tall someone is. Goals are being able to capture that into something that can be modified or measured or graded against.

• Make sure that when you are consuming the data, that the data is accurate.

What do I do with the data?

• Graphing and plotting data, and making sense of what you're seeing is called sensemaking. It is a very important process because different perspectives on how you see and visualize data can alter business outcomes and what you need to do to change the application in response to different events. Try to avoid using templates for high-level business objectives because this might fail to cater to individual apps' specific needs and features, and might be a sign that your company is developing the same application.

• There's different things, like median, mean, mode, percentiles, etc. please do not average percentiles that does not make sense. Averages are so-so, depends on what you are trying to measure. Percentiles might be misleading so be careful on what you are actually measuring and how it is actually impacting customer experience, cite video about 99.99% percentile video about the issues with that and the 20 HTTP request scenario.

• "I"m trying to measure how many users use the application", what does "use" mean, does it mean login? If so, this is usually straightforward. Make sure to account for users logging in multiple times for example.

• "I'm trying to measure some performance of something" ok this might be a bit complicated. Is it measuring from the user's perspective, and then the percentile is over the user's session? For example, the user is clicking on things, and one of their interactions was very slow. Are we aggregating the 90th percentile per user, and then the percentile of that, or aggregating it across all feature interactions? The latter is not as useful, because a single interaction could cause a poor user experience, and it doesn't discern between a single user having a bad time (but used the application a lot), versus many users having a poor experience.

• Performance regressions, web-vitals for front-end, etc.

• Monitoring the actual CI/CD pipeline itself could be useful, for example, if the runner is using more and more RAM, or more and more disk space and might be getting full soon, or it is taking longer and longer to complete (thus compromising the fast feedback loop.) The pipeline is just a server so I'm wondering if regular monitoring tools would apply. If it's slow then it might be using just a single CPU, or too much network, etc.

• Sample size is important, and provides confidence in estimates (use effect size.) Using heuristics are unlikely to be reliable and are difficult to compare over time.

• [https://www.youtube.com/watch?v=Y5n2WtCXz48]{.underline} 12% increase in revenue with faster load times, might be getting a bit off topic...

• For other data, such as goals or diagnostics, you'd typically want to perform sensemaking on them. There are many tools that can connect to your database (e.g., ClickHouse) and can visualize the data. Visualizing data is a way where you can generate insights on the data and be able to do stuff with it. For example, if the performance of a particular part of your application is slow, then you can optimize engineering efforts to improve that piece.

Conclusion

• As we embark on the journey of continuous integration and delivery, monitoring remains our guiding star. It's not about collecting vast amounts of data but making sense of it. With a strategic approach to monitoring, teams can ensure software products that are not just functional but also resonate with the users' needs and preferences.

How do I get the information from my monitors and how do I set up monitoring and logging? How do I associate the logs with a specific application version?

+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | The section on "Monitoring and Feedback in Continuous Deployment" is comprehensive and well-structured. It offers readers insights into various tools, best practices, and methodologies related to monitoring in CI/CD. Here are some suggestions and observations: | | | | 1. Introductory Transition: You could introduce the topic with a sentence or two, setting the context for the reader. For instance: | | | | - "Continuous Deployment relies heavily on effective monitoring and feedback mechanisms to ensure software performance and stability. Let's delve into the specifics of setting up and managing these mechanisms." | | | | 2. Goal of Monitoring: | | | | - Consider highlighting the dual aims: "The primary aims of monitoring are to ensure application performance and reduce or eliminate ambiguity during troubleshooting." | | | | 3. Dashboard Types: | | | | - It might be worthwhile to mention that dashboards are visual representations of data, making them indispensable for rapid diagnostics and decision-making. | | | | 4. Design Considerations: | | | | - Add a point on responsiveness. With the increase in mobile use, ensuring that dashboards are mobile-friendly and adapt to different screen sizes can be crucial. | | | | 5. Performance Monitoring: | | | | - Mention other performance metrics like Disk I/O, Network bandwidth, etc., to give a more rounded view. | | | | 6. Logging Considerations: | | | | - You could also highlight the need for secure logging. Certain data should never be logged (like passwords, personal user details, etc.) due to security and privacy concerns. | | | | 7. 5 W's of Logging: | | | | - Great touch here. This will be incredibly useful for developers and sysadmins alike. | | | | 8. Blue-green database deployment strategies: | | | | - It feels a bit out of place since you provided a detailed section before and after it. Consider expanding on it or positioning it in a more appropriate section or context. | | | | 9. On trust and automation: | | | | - The content here is dense and rich. To enhance readability, consider breaking down the longer sentences. | | | | - Highlight the importance of human oversight. Even with automation, human involvement remains critical for edge cases and unforeseen circumstances. Emphasize that automation is a tool, not a replacement for human judgment. | | | | 10. Citations: | | | | - Ensure that any referenced content is appropriately credited, as you've done. It adds credibility and depth to the material. However, depending on the medium of this content (e.g., a book, an online course, etc.), you may need to follow specific citation styles or consider hyperlinking directly to the source if it's a digital medium. | | | | 11. General: | | | | - A few headings seem to end with URLs (e.g., Logging and Log Management (ipfs.localhost)). Ensure that these URLs are meant to be there or if they were placeholders that should be updated. | | | | - Consider providing some practical examples or case studies. Real-world scenarios or illustrations can help ground the theoretical information. | | | | Overall, you've done an excellent job at consolidating a vast amount of information into a coherent and informative piece. The content covers the essentials of monitoring and feedback in CI/CD thoroughly, making it a valuable resource. | +======================================================================================================================================================================================================================================================================================================================================================+ +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

1. Goal of Monitoring:

The aim is to reduce or eliminate ambiguity when diagnosing incorrect functionality. It's important to log strategically, showing a clear execution path but ensuring it doesn't excessively slow down the application due to data storage concerns.

What should I add to the code to do monitoring? Where do I monitor?

Monitoring software/tools

+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Certainly! Monitoring and observability are critical components of the CI/CD pipeline, especially in production environments. Here's a list of some popular CI/CD software monitoring products: | | | | 1. Datadog: A cloud-based monitoring and analytics platform that allows for full-stack observability by integrating with numerous platforms and services. It offers real-time performance dashboards, end-to-end tracing, synthetic monitoring, and log management. | | | | 2. Prometheus: An open-source system monitoring and alerting toolkit originally built at SoundCloud. It's now a standalone open-source project and maintained independently of any company. It integrates well with the container orchestration system Kubernetes. | | | | 3. New Relic: Provides insights into application performance, infrastructure monitoring, and customer experience. The platform offers a suite of products that track various aspects of applications and infrastructure health. | | | | 4. Splunk: Known for its powerful log aggregation and search capabilities, Splunk has expanded its capabilities to offer infrastructure and application monitoring with its Splunk IT Service Intelligence (ITSI) and SignalFx products. | | | | 5. Elastic Stack (ELK Stack): Comprises Elasticsearch, Logstash, and Kibana. It's widely used for searching, analyzing, and visualizing logs in real-time. | | | | 6. Grafana: An open-source platform for monitoring and observability. Grafana allows users to create, explore, and share dashboards from multiple data sources, including Prometheus, Graphite, and InfluxDB. | | | | 7. Dynatrace: A software intelligence platform that offers application performance monitoring (APM), artificial intelligence for operations (AIOps), cloud infrastructure monitoring, and digital experience management. | | | | 8. AppDynamics: Acquired by Cisco, AppDynamics is an application performance management (APM) and IT operations analytics (ITOA) company. It provides real-time monitoring of applications and infrastructure. | | | | 9. Sentry: An open-source error tracking tool that helps developers monitor and fix crashes in real-time. It's especially useful for identifying issues in code post-deployment. | | | | 10. Raygun: Provides error and performance monitoring for software applications. It helps developers diagnose issues in their applications by providing detailed error diagnostics and performance timing information. | | | | 11. Honeycomb: An observability platform that allows for high-cardinality data exploration, helping developers understand and debug production issues. | | | | 12. LightStep: Focuses on tracing and is particularly optimized for microservices and serverless architectures. | | | | It's worth noting that the best monitoring solution often depends on the specific requirements of the organization, the existing tech stack, and the nature of the applications being monitored. Many companies use a combination of several tools to achieve full-stack observability. | +==========================================================================================================================================================================================================================================================================================+ +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

2. Dashboard Types:

There are generally two types:

Diagnostics Dashboards: These display data without context, such as memory usage or heart rate. They offer a snapshot without specific goals.

KPI Dashboards: These are goal-oriented, showcasing metrics like Monthly Active Users (MAU), Daily Active Users (DAU), customer behavior in an app, or success rates for particular scenarios.

3. Design Considerations:

Maintain minimalism, avoiding unnecessary decorations that could clutter the dashboard.

Collect relevant data, understanding the significance of metrics like averages, percentiles, and exceptional cases.

Prioritize end-to-end (E2E) metrics that mirror user experience, rather than an aggregate of smaller, potentially unrelated metrics.

4. Metrics to Consider:

Focus on higher-level metrics like those from the web-vitals library for web applications to better reflect the user experience.

While HTTP-based metrics are helpful for diagnosis, they may not always be indicative of the overall customer experience.

5. Graphing Data Sources:

There are primarily two categories:

Diagnostics: More developer-centric, they might include metrics like memory usage.

KPIs/Scenario Metrics: More user-focused, they show how users interact with a feature, for instance.

6. Performance Monitoring:

CPU usage can be an indicator, but it's essential to pair it with end-user experience metrics to get a holistic view.

Consider utilizing cloud providers for scalability and fault tolerance. Robust monitoring tools should alert immediately if there's a performance issue, possibly via third-party software to ensure redundancy.

7. Logging Considerations:

Log managers manage, tabulate, and graph logs but don't instruct on what or where to log.

Developers should create clear, concise log messages that provide essential debugging information.

Also important to know what and when to log, and what to include in the log messages.

Assigning priority levels to different logs is crucial. Telemetry is typically what's logged, with different types categorized by importance.

[Logging and Log Management (ipfs.localhost)]{.underline}

In general, logs should provide insights into:

- What Happened?

Provide appropriate details. Merely stating "Something happened" is not particularly useful.

- When Did It Happen?

Include timestamps. If relevant, specify when the event started and ended.

- Where Did It Happen?

Specify details such as the host, file system, network interface, etc.

- Who Was Involved?

Identify the user, service, or system entity.

- Origin:

Where did the entity come from?

These key points represent the "5 W's of Logging". They have been borrowed from disciplines like journalism and criminal investigation, among others.

For a more comprehensive understanding, it's beneficial to know:

- Additional Information:

Where can one find more details about the event?

- Certainty:

How confident are we that the provided details accurately represent the event?

- Impact:

What or who is affected by this event?

If we were to wish for even more insights, it would be great to know:

- Future Events:

What might happen next based on the current event?

- Correlated Events:

What else happened that might be relevant or important?

- Recommended Actions:

Given the event, what should one do next?

Feature Flags

Precedent

• Likely originated (the word "flag" as used in programming to indicate a state) from [International maritime signal flags - Wikipedia]{.underline}

• [Best of Velocity: Move Fast and Ship Things - Facebook's Operational and Release Processes - YouTube]{.underline} facebook popularized it? 2013 so nope

• [10+ Deploys Per Day: Dev and Ops Cooperation at Flickr | PPT (slideshare.net)]{.underline} 2009

What are feature flags?

Feature flags allow for experimentation and integration, and they are essentially remotely controlled "if" statements. These are both vital to CI/CD because this provides the capacity to release changes quickly and effectively--feature flags can be turned off and on remotely, and also allow for multiple partially developed features to co-exist (i.e., have deferred integration) and testable in a local environment while the rest of the application is available to customers. Feature flags allow for features and other work to be gradually rolled out to customers because they can be controlled server-side, giving you control over which users have the feature flag turned on. For example, you can enable it for certain beta users, user type, geography, ip address, account lifetime, etc.

Feature flags work by fetching a key-value pair (typically a feature name and a boolean) from an HTTP API. For instance, a keypair like "EnableSpecialFeature" set to "true" alters application behavior accordingly. Here's a quick practical breakdown of a very simple feature flag implementation:

Typically, the term "feature flag" and "feature toggle" are used interchangeably, but if your team uses one convention over the other or differentiates them then you will have to check with them. In this text, they will be used interchangeably.

Why would I want to use feature flags?

Advantages of Feature Flags

You've already discussed blue-green deployment strategies. Why wouldn't I just use those instead of feature flags?

• They serve different purposes. With feature flags, you can release new features independent of the deployment pipeline, and multiple features can be released at once. You also have more control over who you release it to, such as specific groups of users or via geographical location, and normally you can turn feature flags on and off much faster than going to another deployment through a deployment pipeline. They also allow hiding in-progress development. They also allow exposing features to certain people, or environments, for example QA to test.

• Blue-green deployments are typically reserved for significant changes such as large-scale infrastructure updates, database migrations, or complete framework shifts, like migrating from React to Angular. This method is especially useful for scenarios where feature flags are not feasible, such as with incompatible frameworks or extensive application refactors. It's standard practice to automate the blue-green process to handle these major changes efficiently, ensuring stability and minimal disruption. This approach is also suitable for smaller updates like package upgrades, ensuring all changes, whether minor or major, undergo the same rigorous deployment process.

You want to use feature flags to incrementally expose (or hide) features that are currently being developed. This will be part of your regular CI workflow. When you're working on a feature, put it behind a feature flag. There is an annotated example below with a code implementation.

+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | A/B testing, also known as split testing, is a method of comparing two versions of a webpage, app, email, or other digital assets to determine which one performs better in terms of user engagement or other predefined metrics. In an A/B test, two or more variants, typically referred to as the "A" and "B" versions, are presented to different groups of users simultaneously. The goal is to assess which variant produces more favorable outcomes, such as higher conversion rates, click-through rates, or user interactions. | | | | A canary release, in the context of software deployment and release management, is a deployment strategy that involves rolling out a new version of software or a service incrementally to a subset of users or servers before making it available to the entire user base. This approach is named after the practice of using canaries in coal mines to detect toxic gases; in a canary release, a small, controlled group of users or systems serves as the "canaries" that help detect potential issues or problems with the new release. | +================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================+ +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Feature flags can be straightforward to implement initially, often managed with just a JSON file downloaded at runtime. However, complexity increases when you need more nuanced control, such as rolling out a feature to only 10% of users. This typically requires passing user IDs or request parameters through a server, which handles the feature list applicable to each request. Relying solely on the client-side for this can complicate debugging and requires deployments for changes, somewhat negating the benefits of feature flags. To maintain clarity and trace issues effectively, it's crucial to version feature flags in correlation with the code, potentially through repository management. This ensures that changes in feature flags are trackable and correlated with the codebase for easier troubleshooting.

• You want to perform A/B testing or canary releases to gather user feedback or performance data before fully deploying a new feature. Undertaking A/B experiments to optimize user experience. For instance, enabling features only for specific user segments like beta testers.

• You need to provide different feature sets to different users or user groups, such as premium or trial users.

• You want to develop and release features independently and maintain a single codebase that serves multiple deployment environments.

• Use an effective branching strategy when:

  • You want to manage and organize parallel lines of development, such as features, bug fixes, or release branches.

  • You need to isolate experimental or unstable code changes from the stable main branch to prevent disruptions in production.

  • You want to ensure that different development teams can work on features simultaneously without interfering with each other's work.

  • You need a systematic approach to merge code changes from different branches into the main branch, ensuring the codebase remains stable and up-to-date.

  • You want to maintain a clear version history and facilitate traceability of code changes.

  • Needing high agility, where if an issue arises with a new feature, it can be quickly turned off without redeploying the entire application.

  • Incrementally transitioning from an older system to a newer one using the strangler pattern. For example, redirecting requests from an old application to a new one in real-time while maintaining user session states.

Purpose and Benefits:

• Enable integration: Allow multiple developers to work on different features simultaneously and integrate them seamlessly.

  • Example: Two developers can work on the same weather app, one adding a Fahrenheit conversion feature and the other implementing multiple location search. Feature flags allow them to integrate their work without conflicts.

  • Important Considerations: Features should be relatively modular to avoid excessive conflicts. Early testing with both feature flags enabled helps identify and address integration issues early on.

• Controlled rollout: Gradually release features to users, enabling A/B testing and mitigating risks.

  • Benefits: Gather user feedback, monitor performance, and minimize the impact of potential bugs.

  • Example: Gradually roll out the Fahrenheit feature to a small percentage of users before enabling it for everyone.

• Graceful degradation: Disable problematic features without impacting core functionality, ensuring a better user experience.

  • Example: If the AI-powered weather prediction feature becomes overloaded, it can be temporarily disabled via a feature flag without affecting the basic weather display functionality.

  • Benefits: Prevents a complete application outage and provides a better user experience even during issues.

• Resource management: Control access to resource-intensive features to prevent overload and optimize performance.

  • Example: Limit access to a computationally expensive AI feature or a limited-capacity service to prevent performance degradation for all users.

  • Benefits: Ensure resource availability for critical features and optimize cost by scaling resources according to actual usage.

• Implementation Details:

• Modular features: Features should be modular and independent to avoid conflicts and simplify feature flagging.

  • Challenge: Highly intertwined features can create complex dependencies, making feature flagging less effective and harder to manage.

• Feature flag management: Use a feature flag service to manage flags, determine availability, and provide default values.

  • How it works: The application queries the feature flag service with relevant parameters (user ID, location, etc.) to determine which flags should be enabled.

  • Benefits: Centralized control, easy updates, and the ability to dynamically adjust feature availability based on various factors.

• Frontend and backend implementation: Feature flags can be implemented on both the frontend and backend, depending on the specific use case.

  • Frontend: Suitable for UI changes, feature toggles, and client-side behavior modifications.

  • Backend: Ideal for API changes, infrastructure adjustments, and managing access to backend resources.

  • Example: Switching between different weather API providers would be a backend implementation, while changing the temperature display would be a frontend implementation.

• Consider data implications: Be mindful of data inconsistencies when different user groups have access to different features.

  • Challenge: Storing and managing data generated by different feature variations.

  • Solution: Implement data migration strategies or use feature flags to control data access and ensure consistency.

• Practical Considerations:

• Complexity management: Avoid excessive feature flags and dependencies to prevent code complexity.

  • Risk: Too many feature flags and intricate dependencies can lead to code that's hard to understand, maintain, and debug.

  • Recommendation: Regularly review and remove obsolete feature flags and strive for simplicity in their implementation.

• Project management: Coordinate feature flag usage with development streams and release plans.

  • Challenge: Independent feature development with feature flags can lead to misaligned user experiences when features are released.

  • Solution: Communicate release plans, consider bundling related features, and ensure consistent user experience across feature variations.

• User experience: Ensure a consistent user experience, even when feature flags are enabled or disabled.

  • Challenge: Randomly enabled/disabled features can confuse users.

  • Solution: Design features with feature flag toggling in mind, communicate changes clearly, and avoid abrupt transitions.

• Examples:

• Frontend: Switching between Celsius and Fahrenheit display in a weather app. This could involve duplicating components, changing rendering logic, and updating the UI based on the feature flag's state.

• Backend: Testing a new weather API provider without impacting user experience. This might involve shadow requests to the new API, comparing responses, and gradually shifting traffic once the new API proves stable.

• Resource management: Limiting access to a computationally expensive AI feature. Feature flags can control access based on user subscription level, time of day, or overall system load.

Limitations of Feature Flags

• Increased Complexity: Feature flags can lead to exponential complexity. For instance, with 10 feature flags, there are 2^10 (1024) potential combinations, complicating bug reporting. It might not be clear from the QA's side what to test and such. It's essential to consider all flags when logging issues for reproducibility. It also makes testing more complex. You should log all feature flags and their status on application crash so that it can help you to debug. PCA can be helpful as well for training an ML model to find which flags are likely causing issues (i.e., correlate it with bad output and the list of flags.)

• Inadequate Concealment: Feature flags can't truly hide functionalities. Features shielded behind flags, like those for an enterprise version, could be accessed and enabled client-side. This is also a security issue, if, for example, the code is insecure or contains secret product names that have not yet been released. You can consider transmogrifying your code (e.g., for websites) as this discourages people from snooping into it, but cannot stop it completely.

• Dependency on Default Values: If the feature flag server fails, default values must be set, potentially revealing or hiding features inadvertently. Make sure to set defaults correctly. For example, the feature flag server is down or is not responding, or the client's device is slow.

• Potential for Overuse: Excessive reliance can lead to "death by a thousand cuts," resulting in high cyclomatic complexity and increased testing overhead. Some changes, like OS upgrades, or large refactorings can't be toggled with feature flags.

• Mitigation Strategies:

• Server-Side Rendering (SSR): Evaluate feature flags server-side so clients only receive relevant code. This will hide the code on the server-side so that it is only delivered to clients should they have that feature enabled.

• Monitoring Usage: Track feature flag usage through telemetry, although it's not entirely reliable.

• Semi-Long Lived Branching: Merge main branch changes into semi-permanent branches, creating multiple build artifacts. Deploy certain branches, like testing ones, to designated environments, especially for sensitive features.

• Code Obfuscation: Embed feature flags in the code and then obfuscate it. This approach, relying on security by obscurity, should be used based on risk tolerance. You can obfuscate code to prevent feature flags from being revealed, but ultimately it is in the source code. You can also not add the feature flights to the clients if they should be disabled, and then the client will assume that they are disabled (based on internal programming.) This can help prevent the release of feature flight names which might make people aware of some of the features.

Here's a straightforward example of using feature flags, emphasizing the importance of version control to track when flags are enabled. Initially, create a repository named 'feature flags' and add a flags.json file. Edit this JSON file to deploy changes, ensuring it is well-formed, possibly using a schema for validation. You can manage feature flags for different environments by maintaining separate files.

The deployment pipeline copies the JSON file to an Azure Storage account, which your application accesses at runtime to check the flag status. This method is simple but may not scale efficiently. An alternative is embedding feature flags directly in your code, which requires a separate deployment pipeline for those flags, typically in a continuous deployment style.

However, integrating feature flags with code has its drawbacks, particularly in security. For example, if the storage account is publicly accessible, there's a risk of exposing sensitive code files. It's advisable to use external services to manage feature flags securely and to further research best practices for their implementation.

1. Create a GitHub Repository:

• Go to [https://github.com/]{.underline} and create a new repository (e.g., "feature-flags-azure-storage").

• Initialize the repository with a README file (optional).

2. Local Project Setup:

Clone the repository to your local machine:
git clone https://github.com/your-username/feature-flags-azure-storage.git

cd feature-flags-azure-storage

• content_copyUse code [with caution]{.underline}.Bash

Create a file named flags.json in the root of the repository with the following content:
{

"EnableSpecialFeature": true,

"ShowNewHomepage": false,

"BetaTestingMode": {

"enabled": true,

"users": [ "user1@example.com", "user2@example.com" ]

}

}

• content_copyUse code [with caution]{.underline}.Json

• (Optional) GitHub Actions Workflow (for automatic deployment):

  • Create a .github/workflows directory in your repository.

  • Create a file named deploy-flags.yml (or similar) inside the .github/workflows directory.

Add the following workflow code:
name: Deploy Feature Flags

on:

push:

branches:

• main # Trigger deployment on push to main branch

jobs:

deploy:

runs-on: ubuntu-latest

steps:

• name: Checkout code

uses: actions/checkout@v3

• name: Azure Login

uses: azure/login@v1

with:

creds: ${{ secrets.AZURE_CREDENTIALS }}

• name: Upload to Azure Blob Storage

uses: azure/storage-blob-upload@v1

with:

source_dir: '.' # Upload everything from the root

container_name: 'feature-flags'

storage_account: ${{ secrets.AZURE_STORAGE_ACCOUNT }}

sas_token: ${{ secrets.AZURE_STORAGE_SAS_TOKEN }}

3. Azure Storage Configuration:

• Create Storage Account & Container: Follow step 2 from the previous response.

• Generate SAS Token (Recommended):

  • Navigate to your storage account in the Azure portal.

  • Go to "Containers" -> Select your container ("feature-flags").

  • Go to "Shared access tokens" and generate a SAS token with read permissions and an appropriate expiry time.

4. GitHub Secrets (for secure deployment):

• In your GitHub repository settings, go to "Secrets" -> "Actions".

• Add the following secrets:

  • AZURE_CREDENTIALS: Create a "service principal" in Azure and paste its JSON output here (refer to Azure documentation for details).

  • AZURE_STORAGE_ACCOUNT: Your Azure storage account name.

  • AZURE_STORAGE_SAS_TOKEN: The generated SAS token with read permissions (if using SAS for secure access).

5. Commit and Push:

Commit your changes and push to the main branch:
git add .

git commit -m "Initial setup with feature flags"

git push origin main

• content_copyUse code [with caution]{.underline}.Bash

  If you set up the GitHub Actions workflow, this push will trigger the deployment to Azure Storage.

6. Application Code Integration:

• Endpoint URL:

If using public access (not recommended):
https://your-storage-account.blob.core.windows.net/feature-flags/flags.json

• content_copyUse code [with caution]{.underline}.

If using a SAS token (recommended):
https://your-storage-account.blob.core.windows.net/feature-flags/flags.json?your-sas-token

• content_copyUse code [with caution]{.underline}.

<!-- -->

Code Example (JavaScript):
async function fetchFeatureFlags() {

// ... (code from previous response, step 3 - replace with your actual endpoint URL)

}

// Example usage:

fetchFeatureFlags().then(flags => {

// ... (access feature flags as needed)

});

• content_copyUse code [with caution]{.underline}.JavaScript

Then for example, say if you want to turn the feature on or off or modify something for example.What you can do is just make a change to that feature flag. So just make a commit.Big part requests sends them when it gets deployed through application. Once you had refreshed whatever. Well read that.New.Feature flag.And well.Subsequently changed the output of the application or modifies behavior.And it's also pretty good to log those feature flags that you're using, because it can be especially difficult when you're trying to triage.Different bugs and it becomes very complicated very quickly.

Popular feature flag providers/managers

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1. LaunchDarkly: LaunchDarkly is a widely recognized feature flagging platform that allows teams to manage feature flags and feature toggles with ease. It provides feature management, experimentation, and targeting capabilities to control and optimize feature releases. | | | | 2. Split.io: Split.io is another popular feature flagging and experimentation platform. It offers feature flagging, experimentation, and monitoring tools that enable teams to control feature releases and measure their impact on user behavior. | | | | 3. Optimizely: Optimizely (now part of Episerver) is known for its experimentation and feature management platform. It enables teams to create, manage, and optimize feature flags and experiments to improve user experiences and business outcomes. | | | | 4. ConfigCat: ConfigCat is a feature flag and configuration management platform that helps development teams roll out features gradually and manage configurations across various environments. It supports feature flags, remote configuration, and user targeting. | | | | 5. Flagsmith: Flagsmith is an open-source feature flagging and experimentation platform. It allows teams to create, manage, and control feature flags and experiments to deliver features with confidence. | +========================================================================================================================================================================================================================================================================================+ +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Feature flags vs. branching

• In many cases, it's best to use a combination of both feature flags and effective branching strategies. Feature flags provide the flexibility to toggle features on and off during runtime and manage their rollout, while an effective branching strategy helps manage parallel lines of development and maintain a stable, organized codebase.

• Let's go into more detail about the differences and when to use one over the other.

• Both feature flags and branches can separate parallel lines of development work.

• Feature flags allow for functionality that has already been deployed and is behind a feature flag to be changed at runtime, such as, for example, in production, while customers are using your application. Given a feature flag is just an "if" statement, then this changes the execution flow of the running program. Say for example, I have a feature flag in production, called "enableLogin". If it is off, then the login page won't be accessible to any of the customers. If it is on, then it will be shown. I don't have to do any deployments, releases, or PRs, I just have to change the value of the feature flag in the feature flag manager for this to change. The application then reads the new feature flag and changes its behavior. Changing the feature flag is normally very easy to do, and updates usually occur in the order of seconds.

• Branches allow for parallel lines of development work to be separated at the source-code level, on the business's side inside of the repository, not accessible to customers. For example, say I am working on my feature. I would create a branch, which would allow me to deviate from the other branch (e.g., the trunk.) Or, I'm working on an experiment. Anything I commit (code-wise) to the branch stays on the branch, until I merge it with another branch, which is, effectively transferring the content to the other branch. Branches that are not the trunk are not deployed, thus, my development work stays inside of the repository and does not exist in production. After I merge my branch, the content is then destined for deployment or release, whenever it normally occurs.

• So the big thing is that feature flags allow for integration to occur, as well as experimentation branches.Are.Cause integration to be deferred or never done. They're just independent lines of development work. They technically also allow experimentation. So they allow experimentation because you could theoretically deploy a branch, like say if you had a pull request branch to a local test environment that you could experiment.With your changes.Um, over there as well.

• Think of the metaphor like a physical tree. The branches on the tree aren't part of the main trunk.

• 

• [Tree Free Stock Photo - Public Domain Pictures]{.underline}

What is a feature flag environment?

• An environment is a set of feature flags that are on at a certain time.For example, say if you have a feature and.Um.You.Have some other features as well that are all part of this preview program. For example, you may want to turn them on on the integration environment so you put them in some sort of like environment.

• This environment can be served to different users, or, you might be able to manually choose it.Say for example you want to get the new users of the application access to this feature flight. So for example you want to.Say, Oh yeah, everyone who's the new user is gonna have the capability to track this new fighter, have access to this new UI or something like that.The reason why you might want to use New Years in this case is because the.The older users who have used the application, for example the users that are been around for a longer by not be as familiar with some of the new UI updates. So if you.Give it to new users who don't know much about the current UI, then you kind of.Squeeze in the bed.You can also do it based on different things like region, and you can do it based on language. Depending on what type of your feature is, you can do it based on user preferences. You can get people to opt in manually as well, see if they want to be like a beta tester for example. So there's lots of different ways that you can do this.

• For example, if you want all of the experimental features, you can toggle on the "experimental" environment, and then that environment has many feature flags enabled.

Feature flag lifecycle?

• How long should I leave it in prod for?It depends on your capacity for risk and how long the feature is considered in preview for stability andHow much complexity that you want overtime So like say if you were to keep it in production first forever, you know it is possible. Now there's a possibility that.You know, it does make things more complicated. You could turn it on. Turn it off so it's exists in the database somewhere and you have to.You know, if it's on for 100% of the people, then you know you might might not really be a feature flag anymore. It's more just like a permanent feature.The other thing is now if the feature flight server goes down for whatever reason.Or you know, you have the capacity to disable it, then it means that you have to kind of make sure that if you plan to disable the feature, you do test with the other features that you currently have active. So it does kind of increase the complexity and make a lot more difficult to debug so.Yeah, try to keep that in mind.

• How can I easily clean up the feature flags, and know which one(s) are still in use? Having a good naming patterns are good because then you can do a Ctrl+F on your code and it won't accidentally match other parts of your code. You can also search across emails/messages, etc.There's some AI tools as well. I'm not sure if I can share some of the internal Microsoft tools with regard to some of the feature flag.There's one by Uber called Piranha which doesn't feature flight cleanup as well. You can do something called the time bomb which will automatically deactivate a feature flag after a certain date. Although it's a little bit have a **** approach, you may want to have some sort of like feature flag lifecycle process that alerts you on a certain.Matrix you want to.To that. So it kind of depends.

Feature flag naming patterns

• Feature flags can quickly cause technical debt if they are not cleaned up. Therefore, it is important to make sure that they are easily identifiable.

• Clear, concise names: Short, descriptive flag names.

• Consistent naming convention: Use standard format (e.g., PascalCase, snake_case).SO1 format could be it starts with the letters F and then has the hyphen or an under score or something like that and then it's.And starts or whatever has next like what the features about and then dash and then I don't know the data something like that when it's needs to be disabled or something like that. So as long as there's like some way where.You can easily just search for it. Basically don't have like a bajillion all over the place.And makes things quite a bit more clear.

• Avoid ambiguous names: Use distinguishable, clear names. Do a Ctrl+F everywhere and make sure that that term for your new feature flag is not being used anywhere, otherwise, it'll be hard to find it later.

• Use action verbs/tense: Start names with 'enable', 'disable', or 'toggle'; consistent tense.

• Full words, not abbreviations: Spell out words for readability.

• Positive flag names: Use positive names (e.g., 'enable_feature').

• Use prefixes for categories: Start flag names with a prefix that indicates the category, e.g., 'payment_feature_newGateway' or 'ui_feature_darkMode'. This allows for regex patterns like payment_feature_.* or ui_feature_.*.

• Use suffixes for status: Add a suffix to flag names that indicates their status, such as '_beta', '_experimental', or '_temporary', e.g., 'feature_newUI_beta'. This allows for regex patterns like .*_beta or .*_experimental.

• Indicate flag types: Specify whether a flag is a kill switch, rollout, or A/B test in the name, e.g., 'feature_newUI_rollout' or 'feature_darkMode_abTest'. This allows for regex patterns like .*_rollout or .*_abTest.

• Use feature or epic identifiers: If your flags are tied to specific features or epics in your project management tool, include the corresponding identifiers in the flag names, e.g., 'F123_feature_newUI' or 'EP01_feature_darkMode'. This allows for regex patterns like F123_.* or EP01_.* to match flags related to a specific feature or epic.

• Separate flags by team or department: Organize flags related to specific teams or departments under a common prefix, e.g., 'frontend_feature_newUI' or 'backend_feature_optimization'. This allows for regex patterns like frontend_.* or backend_.*.

• Incorporate version numbers: Include a version number in the flag name, e.g., 'feature_newUI_v2'. This allows for regex patterns like .*_v2 to match flags related to a specific version.

• Utilize hierarchical naming: Adopt a hierarchical naming structure with categories and subcategories separated by delimiters, e.g., 'category.subcategory.feature_status'. This allows for regex patterns like category\..* or .*\.subcategory\..*.

• Include environment information: If applicable, include the environment (e.g., 'dev', 'staging', or 'prod') in the flag name. This allows for regex patterns like dev_.* or .*_prod to match flags specific to a certain environment.

• Use a date format: If applicable, include the creation or activation date in the flag name using a standardized format, such as YYYYMMDD, e.g., 'feature_newUI_20230401'. This allows for regex patterns like .*_2023.* to match flags created or activated in a specific year.

• Avoid using special characters because they will make matching with regex more difficult, and they may not be able to be used in the code, or, some systems may not be able to accept special characters.

Artifacts, Docker, and versioning

What are artifacts?

• Artifacts are anything that is generated by the build process. This could be files, folders, applications, executables, documentation, Docker containers, etc. Therefore, it is important to clarify the context when someone says "artifacts" because this could refer to many different things. However, in practice, this normally refers to the final applications, and does not usually refer to Docker images (as these are sometimes called containers). Docker containers are still outcomes of the build process/deployment process, therefore, they are in theory considered artifacts. It depends on the context that this is being used.

• Artifacts can be grouped together or considered individually. For example, when a pipeline runs, it generates many artifacts, some are just parts of the build process and some are required for the application to run. Artifacts can be grouped together, i.e., packaged (in a tar file) or can be considered individually. One artifact can contain many sub-artifacts.

• Artifacts can be inputs or outputs to a build system, depending on which way or context it is considered in. For example, if application "A" depends on application "B", then application "B"'s artifacts can be considered as inputs for application "A". Normally in this context, this would be considered a dependency on a set of artifacts.

• Tags in Docker allow easily referring to a specific image. If you want the container to be pushed to a specific registry, then the Docker's tag has to contain part of the registry URL. Tags can be thought of both as an identifier and the desired location for the image. This is most likely because the Docker push command does not take a registry as an argument and thus relies on the container tag to disambiguate the context.

• Say I create an image with "docker build .". I get an image, but there's no repository and no tag. This makes it difficult to determine what version or what thing I am looking at.

+-----------------------------------------------------------------------+ | alex@DESKTOP-7M8V9ET:/dev/shm/getting-started-app$ docker images | | | | REPOSITORY TAG IMAGE ID CREATED SIZE | | | | <none> <none> d49c4d85c3ea 24 minutes ago 269MB | +=======================================================================+ +-----------------------------------------------------------------------+

• I don't know what d49c4d85c3ea is. It could contain anything.

• Therefore, we can use tags to keep track of the images.

+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Tagging images in Docker is a vital part of managing and organizing your images, especially when collaborating or deploying applications. Here's a step-by-step guide on how to tag Docker images, push them to registries, and pull them based on their tags: | | | | ### 1. Building and Tagging a Docker Image: | | | | Firstly, when you're building an image using a Dockerfile, you can tag it right away: | | | | bash | | | | docker build -t [username/imagename]:[tag] . | | | | | | | | - \[username/imagename\]: The name of the Docker image (often prefixed with a username or organization name). | | | | - \[tag\]: The tag for the Docker image (e.g., latest, v1.0, development, etc.) | | | | For example: | | | | bash | | | | docker build -t myuser/myapp:v1.0 . | | | | | | | | ### 2. Tagging an Existing Image: | | | | If you have an existing image that you'd like to tag or retag, you can use the docker tag command: | | | | bash | | | | docker tag [source_image]:[source_tag] [username/imagename]:[new_tag] | | | | | | | | For example, to retag an existing myapp:latest image to myapp:v1.0: | | | | bash | | | | docker tag myapp:latest myuser/myapp:v1.0 | | | | | | | | ### 3. Pushing Tagged Image to Docker Hub: | | | | Before pushing, ensure you're logged into Docker Hub (or another Docker registry): | | | | bash | | | | docker login | | | | | | | | Then, push your tagged image: | | | | bash | | | | docker push [username/imagename]:[tag] | | | | | | | | For example: | | | | bash | | | | docker push myuser/myapp:v1.0 | | | | | | | | ### 4. Pushing to Other Registries: | | | | If you're not using Docker Hub, but another registry like Google Container Registry (GCR), Amazon Elastic Container Registry (ECR), or any other, your image name (and tag) will usually include the registry URL: | | | | bash | | | | docker tag myapp:latest registry-url/myuser/myapp:v1.0 | | | | docker push registry-url/myuser/myapp:v1.0 | | | | | | | | ### 5. Pulling a Tagged Image: | | | | To pull an image based on a specific tag: | | | | bash | | | | docker pull [username/imagename]:[tag] | | | | | | | | For example: | | | | bash | | | | docker pull myuser/myapp:v1.0 | | | | | | | | If you don't specify a tag, Docker will usually default to the latest tag: | | | | bash | | | | docker pull myuser/myapp | | | | | | | | ### Tips: | | | | - It's good practice to use meaningful tags. Common tags include version numbers (v1.0, v1.1), development stages (dev, prod), or even Git commit hashes for granularity. | | | | - Keep in mind that while the latest tag might sound like it represents the most recent version of your image, Docker does not enforce this. The latest tag is simply the default tag if no tag is specified. Therefore, it's always recommended to be explicit with your tags to avoid confusion. | | | | - Remember, each time you change and retag an image, you'll need to push the newly tagged image to your registry if you want to share or deploy it. | +==============================================================================================================================================================================================================================================================================================================+ +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

• Let's tag it with "docker tag d49c4d85c3ea my-app:v1.0". The resultant images list now shows our container, with our version:

+-----------------------------------------------------------------------+ | alex@DESKTOP-7M8V9ET:/dev/shm/getting-started-app$ docker images | | | | REPOSITORY TAG IMAGE ID CREATED SIZE | | | | my-app v1.0 d49c4d85c3ea 25 minutes ago 269MB | +=======================================================================+ +-----------------------------------------------------------------------+

• If I make some changes and rebuild the image with "docker build .", then I will get another untagged image,

+-----------------------------------------------------------------------+ | alex@DESKTOP-7M8V9ET:/dev/shm/getting-started-app$ docker images | | | | REPOSITORY TAG IMAGE ID CREATED SIZE | | | | <none> <none> 0e3996fbe4ca 3 seconds ago 269MB | | | | my-app v1.0 d49c4d85c3ea 26 minutes ago 269MB | +=======================================================================+ +-----------------------------------------------------------------------+

• Instead, I can pass the "-t" argument and tag it immediately. This is helpful because you may have multiple images, and so having many untagged at once can cause a bit of confusion. It is also more efficient and makes sure that you don't forget to tag it.

• Now, we have an image that contains the things that your application needs to run. How do we push it to a container registry, where it can be built and published using a CI pipeline?

• NOTE: when you are building images locally, they might contain cached layers. This is a useful property which makes building the containers faster. However, some commands may not be idempotent. For example, running apt-get install curl may install the latest version of curl that is in your package repositories. Depending on how you have your Dockerfile set up, it might be referring to a cached layer which might be outdated. Also, CI runners are unlikely to use cached layers, which is why you might be getting different results when building locally. Therefore, you may consider doing some uncached builds, or, making sure that the items in the steps cannot change by using specific versions of the software.

• Let's go back to publishing it to a container registry.

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Azure Container Registry (ACR) is a managed Docker container registry service used for storing private Docker container images. To publish your Docker image (myapp:v1) to ACR, follow these steps: | | | | ### 1. Prerequisites: | | | | - Ensure you have the azure-cli (Azure Command-Line Interface) installed. | | | | - Ensure you have Docker installed. | | | | ### 2. Authenticate with Azure: | | | | Login to your Azure account: | | | | bash | | | | az login | | | | | | | | A browser window will open asking you to sign in to your Azure account. | | | | ### 3. Create an Azure Container Registry (if you haven't already): | | | | Replace myregistry with a unique name for your registry, and myresourcegroup with the name of your Azure resource group: | | | | bash | | | | az acr create --resource-group myresourcegroup --name myregistry --sku Basic | | | | | | | | You can choose different SKUs (Basic, Standard, or Premium) based on your needs. | | | | ### 4. Login to ACR: | | | | Before you can push an image, you need to authenticate Docker to the Azure Container Registry: | | | | bash | | | | az acr login --name myregistry | | | | | | | | ### 5. Tag Your Image with the Full ACR Login Server Name: | | | | To push an image to ACR, it needs to be tagged with the full ACR login server name. | | | | First, retrieve the login server name: | | | | bash | | | | az acr list --resource-group myresourcegroup --query "[].{acrLoginServer:loginServer}" --output table | | | | | | | | Once you have the login server name (something like myregistry.azurecr.io), tag your image: | | | | bash | | | | docker tag myapp:v1 myregistry.azurecr.io/myapp:v1 | | | | | | | | ### 6. Push the Image to ACR: | | | | Now you can push the image to your Azure Container Registry: | | | | bash | | | | docker push myregistry.azurecr.io/myapp:v1 | | | | | | | | ### 7. Verify: | | | | You can verify that your image was successfully pushed by listing the images in your ACR: | | | | bash | | | | az acr repository list --name myregistry --output table | | | | | | | | And to see the tags for a specific image: | | | | bash | | | | az acr repository show-tags --name myregistry --repository myapp --output table | | | | | | | | You should see v1 in the list of tags for myapp. | | | | ### 8. Optional - Logout from ACR: | | | | After you've finished working with ACR, it's a good practice to log out: | | | | bash | | | | az acr logout --name myregistry | | | | | | | | That's it! Your myapp:v1 image is now published to your Azure Container Registry. Whenever you want to deploy or run this image from the registry, you'll pull from myregistry.azurecr.io/myapp:v1. | +===============================================================================================================================================================================================================+ +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

• Ok, so I have tagged a container. How do I tag the associated source code?

• Git Tags don't do anything on their own; they are not capable of creating a release. The CI or CD runner has to look at the tags and do useful work. This normally occurs when a new tag is pushed.

• Git Tags are one way of tracking releases and offer a provider-agnostic way to check out the code at a specific version. There are many ways to track releases, and sometimes tracking must occur at multiple steps. In this case, there must be tracking at the source code level to make sure that one understands which source code is being released. There may be tracking at the user story or task level to understand which task(s) were part of the release, or QA test plans.

• When creating a new release, and the task(s) are not yet done, put them under the next version that has not yet been released.

• Tags do not change the source code on their own. For example, if your application displays its version in its "About" dialog, this won't change if you tag the release. Therefore, you may want to change the version number in the application before or when you tag the release. This can usually be done via automation and the version number for the application might exist in one of the application's configuration files.

• How do I know when a tag has been pushed or created? I would like to run a script in this case (or kick off another pipeline.)

  • This depends on if the tag is annotated, as the commands will differ. I recommend adding in a manual override, as there will be situations where you may need to delete existing tags or rewrite them because of mistakes or exceptions to the procedure.

  • If you are using a monotonically increasing or random version for each application (e.g., evergreen), then I recommend that this process is automated. If you are using semver, then you may want to consider doing releases manually. It should be very easy to do.

  • It can be complex to manage and create tags when releasing software because it may require knowledge of bash scripting, which people might not be familiar with. It also has different programming paradigms.

  • Normally, you'd want to kick off a release when there is a new tag pushed, and the tag is merged. There are a long tail of exceptional situations, such as two tags being pushed at the same time, tags being deleted, merges, etc. that makes things more complex.

  • First, you'd want to figure out your release strategy before your tagging strategy. Tagging strategy is just a technical implementation of your release strategy.

• Some software allows creating a release manually.

• The issue is, if I am using SemVer for example, how do I automate the tagging process? In this case, there are many tools to show if the tag will be backwards not compatible or not, but SemVer usually requires human intervention because "major" changes are subjective. In this case, releases would still be manually initiated but the process itself would be automated. There are some tools to automatically notify of API breakage, but this would depend on the type of library that you are building and whether there exists a tool for this. It cannot detect all changes, normally, only changes to the public API.

• [java - CI - what to do with old versions? - Stack Overflow]{.underline}

• [c# - Automated Deployment using CI server - Stack Overflow]{.underline}

• [Build versioning in continuous delivery - Stack Overflow]{.underline}

• [Continuous Delivery for multi component Project - Stack Overflow]{.underline}

• What does it mean when a container is generated every time I merge code?

  • Depending on the CI setup, CI might be linked to CD, which means that a deployment is automatically made on every push. Therefore, the CI system might generate the docker container using the Dockerfile included in the repository. The container is usually pushed to a registry after it has been created. [How to use Docker to make releases? - Stack Overflow]{.underline}

• [git - When to create a branch from tag? - Stack Overflow]{.underline} useful to know that each application has a version, although it may not be released to the public. For example, intermediate versions. This would be tedious to do manually.

+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Certainly! Let's delve deeper into the common themes and identify their sub-themes (stackoverflow): | | | | 1. Docker and Image Tags: | | | | * Tagging Process in Docker: | | | | * Example: "While creating a docker-compose file, how is the image section written or what value goes for...?" | | | | * Release Using Docker: | | | | * Example: "How to use Docker to make releases?" | | | | * Tagging and Cloud Integration: | | | | * Example: "I have a CI/CD pipeline configured where Google Cloud Build automatically builds containers from code pushed to a GitHub repo if tagged with a specific." | | | | 2. Tagging Conventions: | | | | * Standard Naming vs. Custom Conventions: | | | | * Example: "Using CI/CD approach, Dev pipeline is creating artifacts with name convention, something like: 1.0-23-SNAPSHOT.jar." | | | | * Link between Tagging and Versioning: | | | | * Example: "This isn't great, since a- I now commit to master, and b- automated CI/CD will grab the previous commit and release it, but it has the old version number." | | | | 3. Automation & Integration: | | | | * Integration with CI/CD Tools: | | | | * Example: "I am testing a Jenkins CI/CD method. There is a single git repo with a Master branch, a QA branch, and a Dev branch." | | | | * Automated Image Updates: | | | | * Example: "kubernetes imagePullPolicy:Always is not pulling image automatically." | | | | * Handling Tagging in Cloud Platforms: | | | | * Example: "Do I need to change the tag on each container with the intended project name before updating the project name itself? Is this handled automatically by GCP?" | | | | 4. Versioning Concerns: | | | | * Auto-Generation of Version Numbers: | | | | * Example: "How can I auto-generate the version number from the git tag?" | | | | * Issues with Committing to Master: | | | | * Example: "I now commit to master, and b- automated CI/CD will grab the previous commit and release it, but it has the old version number." | | | | From this breakdown, it's evident that while tagging and versioning might seem straightforward, they entail nuanced complexities. These complexities amplify when combined with modern tools and platforms, especially in the CI/CD paradigm. As CI/CD seeks to automate software delivery processes, understanding and properly managing these nuances becomes crucial. | +===========================================================================================================================================================================================================================================================================================================================================================================+ +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Integrating Artifact Repositories with CI/CD pipelines

• Package Manager Dependency: Your choice depends on the package manager you're using. For example, C# uses the NuGet package manager, so it would have to be a "restore" step in the pipeline to get the packages from the repository.

• Authorization: Connect to your package repository, often using API keys, credentials, a service connection, or an identity. If you are using your CI/CD providers package manager, it will usually have steps on how to connect to it.

• Local Testing: Before using CI, test the setup locally, potentially using your IDE for assistance.

• Note! Theoretically, artifacts can also be re-generated which would mean that there isn't a need for artifact repositories (i.e., just build the code again.) However, this process is time-consuming and error-prone because the build-tools are usually not version controlled, which means that a small difference in the build tools will cause the outputs to be different. If the output changes by one bit, it does not necessarily mean that program behavior is impacted. However, this means that the artifact is no longer the same, thus opening up the door for potential exploits/vulnerabilities/security issues.

Artifact tracking and naming

• Another issue is: when you have an artifact, how do you trace it throughout its entire lifecycle? For example, say it is in QA. How do you know it is in QA?

• Artifacts are generated at several points during the build process, are generated during non-customer pipeline runs, and during testing. How do I track which one(s) are being used by the customer?

• How do I name artifacts? Organization-module name-revision Repository Layouts JFrog Artifactory Documentation Reader JFrog Help Center.

• When is a version assigned to an artifact?

• Sometimes, the CI or CD runner will assign build numbers to the artifacts.

• An artifact might have a lot of metadata associated with it, such as build numbers, versions, revisions, dates, etc.

• Usually, versions are assigned when doing a release, or they are assigned automatically through the build process, and then whichever version is released, then its version is recorded in the release log.

• A version might exist as a floating version. For example, the marketing material might say "Version 5", when, in fact, there are many updates to that version, such as 5.0.1, 5.2, etc.

• It is also possible to give other developers evergreen versions of the artifact if they are injected at runtime. For example, if they are not bundled with the application (and thus fetched from a remote server), for example a JavaScript payload, then this can ease distribution for multiple clients. You would want to make sure to capture sufficient telemetry, record which version(s) are currently in use so that you can associate error telemetry.

Artifact maintenance

• When you try to maintain artifacts, there are several issues that arise. It might be unclear which versions of the application you are trying to keep and you might keep too many versions. This can cause confusion, especially if you use manual dependency management (although dependency managers can usually automatically choose which version is necessary.) It can be a financial cost as well, and a potential liability if there are too many copies of your application stored everywhere. It increases storage costs as the applications are not necessary to be stored and will never be used. Recall that artifacts are only the essential information that your application needs to run.

• By default, typically, artifacts are stored for 30 days on most providers.

• The other issue is: given that we have artifacts that have been deployed to customers, when do we delete them? After seven years? We might need them again depending on the level of support. If unsure, I would recommend keeping them, because it could be very complex to recreate the artifact from scratch.

• There are usually ways to specify retention policies with your artifact manager.

• When the artifacts are no longer useful, then they can be decommissioned. This is where the artifact managers come into play. They are able to track downloads over time, and might be able to track it down to specific pipelines. This helps you understand where the artifacts are being used. You can also selectively deprecate different versions, which will make it so that application developers cannot use that specific version (unless, of course, it is cached on their machine.)

• When an artifact is deprecated, it might be possible to mark it as deprecated in the dependency manager, and in some cases not make it available for download. You should send sufficient communication to relevant stakeholders regarding its deprecation, including its replacement, when it will be removed, ramifications of what will happen after it is removed, its impact, and who to contact if there are questions. Sometimes, if the artifact is not being used much, you might be able to deprecate it without notifying others.

<!-- -->

• Stuff

  • This can be made more complex if operations have to be performed on those tags. For example, incrementing a tag or determining if one tag is before another tag, for example, is "v1.0-beta" before "v1.0-dev"? For example, incrementing a tag requires knowing what the last tag was, and then adding something to it (or incrementing it.)

  • If you want to tag your releases with branch names, or to associate it with branch names, then consider slugifying the branch name. This is because Docker image names have a restricted character set as to how they can be named.

  • In order to have a reproducible build environment, you have to have enough information about the environment to make it reproducible, such as versions, inputs, their checksums, hardware, etc. Any small change in any part of the software chain can cause the artifacts to be non-reproducible because the tooling is very complex, and has dependencies on other parts of the build process. One way to do this is through Dockerfiles, which are a set of instructions that contain the specific versions of tools that you use to build your application. Because it runs in an isolated environment, this means that you can run multiple conflicting copies of other dependencies on your machine and it will not interfere with the Docker container.

Blue-green database deployment strategies

• See "Refactoring Databases" book.

Chapter 4: The Crucial Role of Testing in CI/CD

Introduction: Why Test?

At its heart, software testing is about understanding the extent to which an application can reliably achieve a user's goals. It's a holistic interpretation, providing confidence not just that the code runs, but that it runs correctly and effectively for its intended purpose. However, it's crucial to remember that tests are only as good as the effort you put into them.

While often perceived primarily as a process for finding defects or bugs, testing is a broad discipline encompassing multiple facets of software development. These include quality, usability, performance, and more. Some aspects of testing fall into the category of "checking" – operations, often automated, that verify if specific, known conditions still hold true. Think of these as demonstrations: things we believe to be true, which a computer programmatically verifies.

It's an illusion to think software can be completely bug-free. Humans make mistakes, the libraries we rely on aren't perfect, and even hardware can fail. Therefore, the focus of testing isn't the impossible goal of eliminating all bugs. Instead, it's about ensuring the software works well enough to meet user needs and achieve business objectives, managing risk rather than pursuing theoretical perfection.

Testing often refers to the structured evaluation of software against predefined criteria, frequently using automated tests. When developers or teams claim they "don't test," they usually mean they lack a formal testing plan or extensive automation. In reality, even fundamental actions like compiling code or navigating a website after deployment are forms of testing. Any interaction with software, whether by developers during creation or by customers during use, inherently involves testing. If software is never used, and no one notices if it's broken, its value and relevance become highly questionable.

Think of testing like guardrails on a highway. They help ensure traffic stays on the intended path but cannot absolutely guarantee it. Too many rigid guardrails (overly specific or numerous tests) can make it difficult to change the path later – for instance, when refactoring code or adding new features. Conversely, too few guardrails (insufficient testing) make it hard to assess the impact of changes, especially in complex systems. Finding the right balance, knowing how "tight" to make your tests, is essential.

In the context of Continuous Integration and Continuous Deployment (CI/CD), automated testing provides a rapid feedback loop. Developers can quickly verify their changes without disrupting the entire system or requiring lengthy manual checks. Tests are typically run automatically before changes are integrated (e.g., on pull requests) and sometimes during the integration process itself. This ensures that tests remain reliable, as a failed test can, and often should, halt a deployment. This efficiency means developers catch errors swiftly, speeding up the overall development cycle, leading to higher-quality products for customers, and freeing up Quality Assurance (QA) professionals to focus on more complex, exploratory, and user-centric testing activities.

Going forward in this chapter, we will often discuss testing in two broad categories: automated testing and manual testing. This is, technically, a false dichotomy, as the lines can blur. However, this distinction is practical for CI/CD because automated tests can be executed by CI/CD runners and contribute directly to the automated pipeline, whereas manual testing requires human intervention. We will use these terms with this distinction in mind. Automated testing is a cornerstone of effective CI/CD, enabling the fast feedback loop that allows developers to confidently introduce changes with reduced risk.

It's also vital to understand that writing tests is not a one-time task. Tests must evolve alongside the application. As features change or are added, corresponding tests need to be created, updated, or sometimes removed. Tests are typically written concurrently with the feature code and should be included in the same pull request (PR) for review. Critically, tests should undergo the same level of scrutiny during code review as the feature code itself.

Testing becomes particularly crucial when a system grows too large or complex for a single developer to effectively reason about the full impact of their changes. There must be some level of testing for any feature; otherwise, there's no verifiable evidence that the feature works as intended, breaking the chain of integrity from requirement to deployment.

Tests, in essence, are designed to keep things working as expected – to maintain invariants. However, software development often involves constant evolution and change. This creates a natural tension: tests aim for stability, while development introduces change. Excessive or poorly designed tests can drag down development velocity. Therefore, a balance must be struck. It's impossible to test code 100%, nor would it be desirable, as it would imply testing for an infinite amount of time and scenarios. The goal is to write useful tests that provide real value. This involves knowing what to test – focusing on critical functionalities, areas prone to change, or aspects that might not evolve frequently but are vital. There's an inherent complexity in many systems that cannot simply be architected away; tests are a key tool for managing this complexity and the interdependencies between modules. Without them, developers would need to perform extensive manual checks or spend inordinate amounts of time tracing code paths – processes that are both time-consuming and highly error-prone.

What is Quality? [Concerning Quality]

Defining "quality" in software is challenging because it's inherently subjective. It's rooted in the alignment between perceived expectations and actual standards or outcomes. Because expectations and standards can shift depending on the context, user, or business need, quality is dynamic.

There's also a degree of ethics involved. Quality implies that a product is offered in good faith, meeting certain implicit or explicit promises. This is particularly important when consumers cannot immediately assess the true quality at the point of purchase. The perceived quality directly impacts the seller's reputation, influencing customer trust and future decisions.

Utility – the product's ability to meet or exceed the functional expectations of its users – is a core aspect of quality. Does the software solve the problem it was intended to solve? Does it fulfill its purpose effectively? Significant deviation from these expectations typically leads to negative perceptions of quality.

Interestingly, the lifetime or perpetual existence of a product doesn't necessarily equate to its quality. A piece of software might solve a specific, time-bound problem and then be retired, yet still have provided immense value during its lifespan. Its quality might even intangibly improve the quality of other processes or products it interacted with. Even deleted software retains the immutable value of the problems it solved in the past. Furthermore, software currently serving no active purpose might hold future value, perhaps for mitigating risks, complying with audits, or being repurposed later. This again highlights the subjective and context-dependent nature of quality.

Testing serves as the mechanism to ensure the product meets these varied expectations. It verifies that the product indeed solves the intended problem and fulfills its purpose for the users and the business.

Writing tests shouldn't feel like a chore, akin to "eating your vegetables." Tests are written because they provide tangible utility. Performing all necessary verification manually is often inefficient and error-prone. Developers need a reasonable level of confidence that their changes haven't inadvertently broken something elsewhere in the application – something they might not even be aware of. In large applications, holding the entire system's complexity in one's head is impossible. Tests provide the necessary safety net and validation mechanism.

A Little History: Fixtures and Mocks

To understand some common testing terminology, it helps to look at its origins, particularly from hardware engineering.

Test Fixtures: The term "test fixture" originates from hardware manufacturing. A physical test fixture was literally a device designed to securely hold a piece of hardware (like a circuit board) in a consistent position for testing. This ensured reliable and repeatable measurements.

In software testing, this concept was adapted. A software test fixture refers to a known, baseline state or environment set up before tests are run. This might involve initializing variables, setting up database records, configuring global states, or preparing other dependencies so that tests can execute from a consistent starting point and easily access the required state.

Mocks: In everyday language, a "mock" is a replica or imitation. In hardware, a mock object might be a stand-in that mimics some, but not all, functionality of a real component. This could be useful if the real component is expensive, rare, or unavailable during testing.

In software development, "mocking" involves creating substitute objects or functions that are called instead of the real ones during a test. These mock objects are created by the developer to simulate the behavior of the real dependency, often in a simplified way. This is useful for isolating the code under test from its dependencies, avoiding the overhead of interacting with real databases, networks, or third-party services, or simulating specific scenarios (like network errors or empty database results) that might be hard to reproduce otherwise. Mocks typically perform less processing than the components they imitate but can be configured to return specific values, accept certain inputs, or verify that they were called correctly.

(Historical timeline omitted for chapter flow, but the concepts are introduced here)

The Role and Purpose of Tests Revisited

Why do we fundamentally need tests? Because systems, and the humans who build them, are fallible. If developers always knew the exact intent and consequences of every change, and could perfectly verify alignment with desired behavior, formal tests might be redundant. The verification would happen implicitly during development.

Tests exist to check invariants – conditions or properties that are expected to remain true. Many tests, especially granular ones like unit tests, implicitly assume that any change causing a deviation from the tested behavior is undesirable. For example, if a function's output changes and a unit test fails, it could signal a newly introduced bug. However, it could also signal an intentional feature change that requires the function to behave differently. The test itself doesn't inherently know the difference; it only knows the previously defined contract has been violated. It provides information, and the programmer must interpret it. This highlights a potential friction point: in fast-moving projects with frequent requirement changes (like early-stage startups), tests might need frequent rewriting, potentially reducing their immediate return on investment.

Tests act as safeguards against unwanted changes, but their effectiveness is limited by the scope and quality of the test coverage and specific test cases. They provide critical information, especially in large systems where it's impossible for one person to fully grasp the ripple effects of their changes. Tests help prevent excessive or unintended change by enforcing known contracts.

This inherent nature of tests – preventing change – means they introduce a trade-off. Tests generally slow down the initial development process (time spent writing and running them) in exchange for increased resilience and predictability, preventing unintended consequences later. Apps naturally evolve due to new feature requests or external factors like security updates and library deprecations, which require refactoring. There's a constant push and pull between the desire for stability (enforced by tests) and the need for change.

Is slowing down development necessarily bad? Not always. It depends on the value derived. While tests add overhead to the initial creation of a feature, they can significantly increase speed in the long term by preventing rework. Fixing bugs caught by tests during development is far cheaper and faster than fixing them after they've reached production and impacted users. The overall effect on development speed depends heavily on factors like how quickly tests run, the risk tolerance of the application, and the quality of the testing strategy itself.

One might argue that even if tests run instantaneously and consume no resources, they still slow down the process because their output (pass/fail information) must be processed and potentially acted upon. If the information from tests doesn't influence decisions or software outcomes, then running them is pointless, as their computation isn't used by customers and their state isn't retained. Therefore, testing inherently introduces a delay because its informational output needs to impact the workflow.

A counter-argument suggests that the mere act of writing tests is valuable, even if failures are ignored, because it forces developers to understand the code better and can serve as documentation. However, to gain that understanding or use it as documentation, one must verify that the test works, which brings us back to needing the information derived from running the test (i.e., knowing if it passed or failed).

Ultimately, tests serve several key purposes:

1. Preserving Intent: Ensuring that changes have the intended effect (e.g., changing a button's color changes only that button, not the page background).
2. Verifying Functionality: Treating the application (or parts of it) as a function that must produce expected outputs or state changes given certain inputs, within acceptable tolerances.
3. Confirming User Goals: Checking if users can successfully complete their intended tasks. It doesn't matter how many low-level API tests pass if the end user cannot achieve their goal.
4. Meeting Business Needs: Ensuring requirements beyond immediate user interaction are met (e.g., auditing requirements, telemetry collection). Customers might not directly care about these, but the business does.
5. Maintaining Established Quality: If prior versions established certain levels of usability, performance, clarity, and relevance, tests serve as a proxy for maintaining these qualities by ensuring the application behaves consistently (within defined boundaries).

Okay, continuing the chapter draft.

Types of Testing in the CI/CD Workflow

While there are many ways to categorize software tests, several types are particularly relevant within a CI/CD context. Understanding their purpose helps in building an effective testing strategy.

Unit Testing

Unit tests focus on the smallest testable parts of an application, often individual functions or methods within a class. They test these "units" in isolation from the rest of the system. The primary goal is to validate that each piece of code performs its specific task correctly according to its design.

Because they operate on small, isolated code segments, unit tests are typically very fast to run. This makes them ideal for inclusion early in the development workflow, often run by developers locally before they even commit their code, and again automatically on every pull request. They provide rapid feedback on the correctness of individual components.

To achieve isolation, unit tests often employ mocks or stubs to replace dependencies (like database connections, network calls, or other functions). This ensures the test focuses solely on the logic within the unit itself, without being affected by the behavior or availability of external systems.

Example: Simple Unit Test (C# using MSTest)

Imagine a simple Calculator class:

// In YourNamespaceWhereCalculatorExists
public class Calculator
{
    public int Add(int a, int b)
    {
        return a + b;
    }
}

A unit test for the Add method might look like this:

using Microsoft.VisualStudio.TestTools.UnitTesting;
// Make sure to reference the project containing Calculator
using YourNamespaceWhereCalculatorExists;

[TestClass]
public class CalculatorTests
{
    [TestMethod]
    public void Add_TwoNumbers_ReturnsCorrectSum()
    {
        // Arrange: Set up the test.
        var calculator = new Calculator();
        int number1 = 3;
        int number2 = 4;
        int expectedSum = 7;

        // Act: Execute the code under test.
        var result = calculator.Add(number1, number2);

        // Assert: Verify the outcome.
        Assert.AreEqual(expectedSum, result, "The sum was not calculated correctly.");
    }
}

This test follows the common Arrange-Act-Assert (AAA) pattern: set up prerequisites, invoke the code, and verify the result.

Unit tests excel at verifying internal logic, handling edge cases (e.g., what happens when input is null or zero?), and ensuring that specific functions meet their contracts. They are particularly useful when dealing with complex algorithms or logic that might be difficult to trigger or observe through the user interface alone. For example, testing error handling for an "out of stock" scenario might be easier with a unit test than by manipulating inventory levels in a full application environment.

However, unit tests are often tightly coupled to the implementation details. Refactoring code, even if the external behavior remains the same, can easily break unit tests, leading to maintenance overhead. Over-reliance solely on unit tests can also lead to situations where individual components work perfectly in isolation but fail when integrated.

Integration Testing

Integration tests take the next step up from unit tests. They verify the interaction between different units, components, or layers of the application. The focus shifts from isolated correctness to ensuring that combined parts work together as expected.

Examples include testing:

• Communication between a service layer and a database.
• Interaction between different microservices via API calls.
• The flow of data through multiple components.

Integration tests often require more setup than unit tests, potentially involving real databases (or in-memory versions), network communication, or interaction with other actual services. Consequently, they tend to be slower to run.

Example: Simple Integration Test (C# with EF Core In-Memory DB)

Consider a UserService interacting with a database via Entity Framework Core:

// Entity and DbContext (simplified)
public class User { public int Id { get; set; } public string Name { get; set; } }

public class AppDbContext : DbContext
{
    public AppDbContext(DbContextOptions<AppDbContext> options) : base(options) { }
    public DbSet<User> Users { get; set; }
}

// Service using the DbContext
public class UserService
{
    private readonly AppDbContext _context;
    public UserService(AppDbContext context) { _context = context; }
    public User GetUser(int id) { return _context.Users.Find(id); }
}

An integration test verifying the service retrieves data from the (simulated) database:

using Microsoft.EntityFrameworkCore;
using Microsoft.VisualStudio.TestTools.UnitTesting;
// Add necessary using statements for your classes

[TestClass]
public class UserServiceIntegrationTests
{
    private AppDbContext _context;
    private UserService _service;

    [TestInitialize] // Runs before each test
    public void TestInitialize()
    {
        // Use an in-memory database for testing
        var options = new DbContextOptionsBuilder<AppDbContext>()
            .UseInMemoryDatabase(databaseName: System.Guid.NewGuid().ToString()) // Unique name per test run
            .Options;
        _context = new AppDbContext(options);
        _service = new UserService(_context);

        // Seed database with test data
        _context.Users.Add(new User { Id = 1, Name = "Alice" });
        _context.SaveChanges();
    }

    [TestMethod]
    public void GetUser_ValidId_ReturnsUserFromDatabase()
    {
        // Act
        var user = _service.GetUser(1);

        // Assert
        Assert.IsNotNull(user);
        Assert.AreEqual("Alice", user.Name);
    }

    [TestCleanup] // Runs after each test
    public void TestCleanup()
    {
        _context.Database.EnsureDeleted(); // Clean up the in-memory database
        _context.Dispose();
    }
}

This test verifies the interaction between UserService and AppDbContext using a realistic (though in-memory) database setup.

Integration tests are crucial for uncovering issues that arise at the boundaries between components, such as data format mismatches, incorrect assumptions about dependencies, or communication failures.

End-to-End (E2E) Testing

End-to-end tests simulate a complete user workflow through the application, from the user interface (UI) down through the various layers (services, databases, external integrations) and back. They aim to validate the system as a whole from a user's perspective.

Examples include:

• Simulating a user logging in, adding an item to a shopping cart, and checking out.
• Making an API request to a specific endpoint and verifying the entire response structure and data, simulating how a client application would interact with it.

E2E tests are typically the most comprehensive but also the slowest and potentially most brittle type of test. They often involve automating a web browser (using tools like Selenium, Cypress, or Playwright) or making actual HTTP requests to deployed environments. Because they interact with the full system, including the UI, changes to layout, element IDs, or underlying service behavior can easily break them.

They are invaluable for ensuring that critical user journeys function correctly and that all the integrated parts truly deliver the expected end-user experience. A failure in an E2E test often indicates a significant problem that would likely impact real users.

Regression Testing

Regression testing isn't a distinct type of test like unit or E2E, but rather a purpose for running tests. Its goal is to ensure that new code changes (features, bug fixes, refactoring) have not negatively impacted existing functionality. Essentially, it aims to prevent "regressions" – bugs reappearing or previously working features breaking.

Any existing unit, integration, or E2E test can serve as a regression test. When a bug is found and fixed, it's common practice to write a specific test (often a unit or integration test) that reproduces the bug. This test initially fails, passes once the fix is applied, and is then kept in the test suite to ensure the bug doesn't resurface later. Running the entire relevant test suite after changes provides confidence that existing functionality remains intact.

Performance and Load Testing

These tests focus on the non-functional aspects of application speed, responsiveness, stability, and resource utilization, especially under load.

• Performance Testing: Measures response times and resource consumption under typical or specific conditions.
• Load Testing: Simulates concurrent user access to see how the system behaves under heavy traffic, identifying bottlenecks and capacity limits.

While standard functional tests might have timeouts, performance and load tests use specialized tools (like k6, JMeter, or Locust) to generate significant traffic, measure precise timings, and collect detailed metrics (CPU usage, memory consumption, network I/O). Changes to code, even small ones, can subtly degrade performance over time. Regular performance testing helps ensure the application continues to meet user expectations and Service Level Agreements (SLAs). These tests are often run less frequently than functional tests, perhaps nightly or before major releases, due to their resource-intensive nature.

Non-functional, User, and Security Testing

This broad category encompasses tests that don't focus solely on whether a specific function produces the correct output but rather on other qualities:

• Usability Testing: Evaluating how easy and intuitive the application is for users. This often involves observing real users interacting with the system and relies heavily on human judgment and feedback. Automated tests struggle here as they lack the concept of intuitiveness.
• Accessibility Testing: Ensuring the application is usable by people with disabilities (e.g., screen reader compatibility, keyboard navigation, sufficient color contrast). Some aspects can be automated, but manual checks are often essential.
• Security Testing: Identifying vulnerabilities and ensuring the application protects against threats like SQL injection, cross-site scripting (XSS), unauthorized access, etc. This involves specialized tools (scanners, penetration testing frameworks) and expertise.
• Exploratory Testing: A less structured approach where testers simultaneously learn about the software, design tests, and execute them, often based on intuition and experience. This human-driven activity is excellent for finding unexpected issues that rigid test scripts might miss.

While CI/CD heavily emphasizes automated tests for speed and consistency, these other forms of testing, often involving manual effort and human expertise, remain critical for delivering a truly high-quality, secure, and user-friendly product.

Testing Frameworks and Tools

To write, organize, and run tests efficiently, developers rely on testing frameworks and tools. Frameworks like JUnit (Java), pytest (Python), Jest (JavaScript), MSTest/NUnit/xUnit (.NET), and Google Test (C++) provide structure and utilities for testing.

Key benefits of using a testing framework include:

1. Structure: They provide conventions for defining tests (e.g., using attributes like [TestMethod] or specific function naming patterns), making tests easier to write and understand.
2. Execution: They include test runners that discover and execute tests automatically.
3. Assertions: They offer built-in functions (Assert.AreEqual, expect(value).toBe, etc.) for verifying expected outcomes.
4. Setup/Teardown: They provide mechanisms (like [TestInitialize] / [TestCleanup] or beforeEach / afterEach) to set up preconditions before tests and clean up afterward, ensuring test independence.
5. Reporting: They can generate reports detailing test results (pass/fail counts, duration, errors), often in formats consumable by CI/CD systems (like JUnit XML). This allows pipelines to track test outcomes, display results, and make decisions (e.g., fail the build if tests fail).
6. Integration: Many frameworks integrate well with IDEs (for easy local running and debugging) and CI/CD platforms.

Tools like Selenium, Cypress, or Playwright focus specifically on automating browser interactions for E2E testing. Others like Postman or REST Assured help with API testing. Mocking libraries (Mockito, Moq, NSubstitute) assist in creating mock objects for unit testing. These tools often work in conjunction with the core testing frameworks. Using established frameworks and tools promotes consistency within a team, leverages community knowledge, and automates much of the boilerplate work involved in testing.

Organizing Your Tests

As a project grows, so does its test suite. Proper organization is crucial for maintainability and efficient execution.

• Location: Conventions vary by language and framework, but common patterns include:
  • Placing test files alongside the source files they test (e.g., myFunction.js and myFunction.test.js).
  • Using a dedicated test directory structure that mirrors the source directory structure (common in Java and C#).
  • Having a top-level tests or spec directory.
• Naming Conventions: Clear and consistent naming is vital. A good test name describes what scenario is being tested and what the expected outcome is (e.g., Add_TwoNegativeNumbers_ReturnsCorrectNegativeSum).
• Grouping/Suites: Frameworks often allow grouping tests into suites (e.g., by feature, type like "unit" vs "integration", or speed like "fast" vs "slow"). This enables running specific subsets of tests. For instance, during local development or on a PR build, you might only run fast unit tests, reserving slower integration or E2E tests for a nightly build or pre-deployment stage. Some advanced test runners can even automatically determine which tests are relevant based on the code changes made.

Good organization prevents test duplication, helps ensure adequate coverage across different functionalities, makes it easier for developers to find and run relevant tests, and simplifies debugging when tests fail.

Okay, let's continue building the chapter, focusing on best practices, challenges, and developing a sound testing philosophy within the CI/CD context.

Best Practices and Challenges in Software Testing

Writing tests is one thing; writing effective tests and managing them within a dynamic CI/CD environment presents its own set of practices and challenges.

Ensuring Broad Coverage While Avoiding Overlap

A good test suite provides confidence by covering the application's critical functionalities. However, simply writing more tests isn't always better. Strive for broad coverage of requirements and user scenarios, but be mindful of redundant tests. Overlapping tests (multiple tests verifying the exact same narrow piece of logic) increase maintenance overhead without significantly improving confidence. Well-organized tests, perhaps structured by feature or user story, help identify gaps and prevent unnecessary duplication. While some overlap is inevitable (e.g., setup steps), deliberate effort should be made to ensure each test adds unique value.

Selectively Running Tests

As test suites grow, running all tests on every single code change locally can become prohibitively slow, hindering the fast feedback loop. Developers need the ability to selectively run tests relevant to their current changes. Most testing frameworks support running individual tests or specific suites. Some modern tools even offer test impact analysis, attempting to automatically determine which tests could be affected by a given code change and running only that subset.

In the CI pipeline, the strategy might differ. Pull request builds often run a faster subset (e.g., unit tests and core integration tests), while post-merge builds or pre-deployment stages might execute the full suite, including slower E2E and performance tests. The key is balancing feedback speed with test thoroughness at different stages.

Understanding Code Coverage (and its Limitations)

Code coverage tools measure which lines or branches of your source code are executed by your test suite, typically expressed as a percentage. It can be a useful indicator, but it's crucial to understand its limitations.

• What it shows: If a section of code has 0% coverage, it means no test executes it. This is a clear signal that part of your application is untested.
• What it doesn't show: High coverage (e.g., 90% or even 100%) does not guarantee the tests are meaningful or that the code is bug-free. It only shows that the code was executed, not that the assertions within the tests were correct or comprehensive. A test could run through code without actually verifying the right behavior.
• The danger of targets: Setting arbitrary high coverage targets (e.g., mandating 90% coverage) can incentivize developers to write trivial or low-value tests simply to "hit the number," potentially making the codebase harder to refactor later due to the sheer volume of tests, some of which might be brittle. It's often unclear what the untested 10% in a 90% coverage scenario represents – was it low-risk boilerplate code, or a critical edge case that was hard to test?

Use code coverage as a tool to identify untested areas, but don't treat it as a definitive measure of test quality. Focus on testing critical paths and complex logic thoroughly, rather than chasing a percentage.

Mutation Testing: A Deeper Look

Mutation testing offers a more sophisticated way to assess test suite quality than simple code coverage. It works by automatically introducing small changes ("mutations") into your source code (e.g., changing a + to a -, > to <, or deleting a line). It then runs your test suite against each mutated version.

• If a test fails, the mutation is considered "killed" – meaning your tests were effective enough to detect that specific change.
• If all tests still pass despite the mutation, the mutation "survives" – indicating a potential weakness in your tests; they weren't specific enough to catch that particular alteration.

A high percentage of killed mutants suggests a more robust test suite compared to one where many mutants survive. However, mutation testing is computationally expensive and often run less frequently than standard tests.

Analyzing and Interpreting Test Results

Tests generate data – pass/fail status, execution time, error messages. Effectively analyzing this data is key. CI/CD platforms often provide dashboards to visualize test results over time. Look for patterns:

• Frequently Failing Tests: Which tests fail most often? This might indicate brittle tests or unstable areas of the application needing attention.
• Slow Tests: Which tests take the longest? Can they be optimized, run less frequently, or parallelized?
• Flaky Tests: Tests that pass and fail intermittently without code changes (discussed more under Anti-patterns). These erode confidence and must be addressed.

Publishing test results (often via standard formats like JUnit XML) allows aggregation and trend analysis. This data helps prioritize fixing problematic tests and identifying systemic quality issues. Remember, however, that a lack of failing tests in one area doesn't automatically mean it's high quality – it might simply lack adequate testing.

Where and When to Run Tests

The placement of tests within the CI/CD pipeline influences the feedback loop and risk mitigation:

1. Locally (Developer Machine): Running fast tests (mainly unit tests) locally before committing/pushing provides the quickest feedback, catching errors before they affect others.
2. On Pull Request (PR): Running a core set of automated tests (unit, key integration) automatically when a PR is created/updated acts as a gatekeeper. Failing tests block merging, preventing broken code from entering the main branch and "keeping the pipeline green" for deployments.
3. Post-Merge (Main Branch): After a PR is merged, a more comprehensive suite (potentially including slower integration tests) might run on the main branch to ensure integration integrity. This build often generates the artifacts used for deployment.
4. Pre-Deployment (Staging/PPE): Before deploying to production, tests (often E2E, performance) might run against a production-like environment (Staging or Pre-Production Environment - PPE) to validate the actual deployment artifact and configuration in a realistic setting.
5. Post-Deployment (Production): Some tests ("smoke tests" or health checks) run against the live production environment immediately after deployment to quickly verify core functionality is working. This is the ultimate validation but carries the risk of impacting real users if not done carefully (e.g., using read-only checks or dedicated test accounts).

Why Run Tests on CI and Locally?

It might seem redundant, but running tests in both environments is crucial:

• Discipline & Oversight: Developers might forget, lack discipline, or only run a subset of tests locally. The CI server acts as an unbiased enforcer, ensuring all necessary tests pass before integration.
• Environment Differences: A developer's machine is rarely identical to the CI environment or production. Tests might pass locally due to specific configurations, installed tools, data, timezones, or OS differences that don't exist elsewhere. The CI server provides a cleaner, more standardized environment, closer to production.
• Comprehensive Testing: CI servers are better suited for running long, resource-intensive tests (E2E, load, performance) that might be impractical locally.
• Clean Builds: CI systems typically build projects from scratch, avoiding issues caused by leftover artifacts or inconsistent state on a developer machine, ensuring repeatable builds.
• Dependency Checks: If a shared library changes, a CI server can potentially trigger builds for all dependent projects to catch downstream breakages early.

When are Mocks Useful?

Mocking shines when you need to isolate the code under test or control its environment:

• Isolating Logic: Testing complex calculations in a shopping cart (e.g., handling discounts, taxes, out-of-stock items) without needing a real UI or database.
• Simulating External Systems: Testing how your login page handles an invalid password response from an authentication service, without needing a live service that might be unavailable or slow. Testing how a search function behaves when the underlying search engine returns no results or throws an error.
• Controlling Difficult States: Verifying how a payment gateway integration handles a scenario where the bank's system is temporarily down – a state hard to reproduce on demand with the real system.
• Performance: Avoiding slow network calls or database queries during fast unit tests.
• Verifying Interactions: Ensuring specific methods on dependencies were called (e.g., checking if a logging service was invoked correctly).

Testing Philosophy and Prioritization

Simply knowing the types of tests isn't enough. You need a coherent philosophy and strategy to guide what, when, and how to test effectively.

Beyond the Pyramid: Test Where it Makes Sense

The "Testing Pyramid" (many unit tests, fewer integration tests, fewest E2E tests) is a popular heuristic. It emphasizes placing tests at the lowest possible level for speed and isolation. While the underlying principle (prefer faster, more focused tests when appropriate) is sound, rigidly adhering to the pyramid's shape can be misleading.

Don't write unit tests just to make the pyramid look right if the real risks or integration points demand integration or E2E tests. Conversely, don't use slow, brittle E2E tests to verify simple algorithmic logic that a unit test could cover instantly.

The critical question is: What are you trying to verify, and what's the most effective and efficient way to do it?

• If you need to validate complex business logic involving multiple components interacting, an integration test might be necessary.
• If you need to ensure a user can complete a critical workflow through the UI, an E2E test is likely required.
• If you need to verify a specific calculation or edge case within a single function, a unit test is probably best.

Track why your tests fail. If 95% of UI test failures are actually due to calculation errors in the backend (as noted in one example), then using slow UI tests for this purpose is inefficient. Add targeted unit or integration tests at the source of the calculation instead. Test at the layer where the potential issue originates and can be most directly verified.

Outcome vs. Process Testing

Consider what aspect of the behavior is important:

• Outcome-Focused: Do you primarily care about the final result, regardless of how it was achieved? Example: Testing if clicking the "Login" button successfully navigates the user to their dashboard. You don't care exactly how the button was rendered or which internal services were called, only that the user-visible outcome is correct. E2E tests often excel here.
• Process-Focused: Is the way the result is achieved critical? Example: Testing if a caching layer is actually being used when retrieving data. Simply checking the returned data isn't enough, as the data would be the same whether the cache was hit or the database was queried directly. You need to verify the internal process (e.g., by mocking the database and ensuring it wasn't called, or by inspecting the cache state). Unit or integration tests with mocking/spying capabilities are often better suited for this. Another example is verifying that specific audit logging functions are called during a transaction, even though the user never sees the audit log.

Understanding whether you're testing the outcome or the process helps select the appropriate test type and level.

E2E Tests: Necessary but Handle with Care

There's sometimes a push to minimize E2E tests because they can be slow and brittle. While reducing unnecessary E2E tests is good, eliminating them entirely is often unwise. They are the only tests that truly verify the integrated system from the user's perspective.

Instead of just reducing their number, focus on:

• Stability: Use reliable selectors, wait strategies, and consider tools designed for robustness (like Cypress or Playwright).
• Scope: Focus E2E tests on critical user journeys, not every single UI element interaction.
• Placement: Run them at appropriate pipeline stages (e.g., pre-deployment) rather than on every commit.
• Optimization: Can they be run in parallel? Can the underlying environment be made faster?

The Confidence Factor

Ultimately, tests are about providing confidence to developers and the business that changes can be deployed safely. Tests should be meaningful and well-designed. A suite full of trivial tests passing gives a false sense of security. Code passing tests doesn't automatically mean the code is correct; it only means it meets the specific expectations encoded in those tests. Well-designed tests, however, significantly increase the probability that passing tests correlates with correct code.

Okay, let's continue drafting the chapter, moving into operational strategies, test management, and tackling common challenges.

Operational Strategies for Testing in CI/CD

Beyond writing individual tests, effectively managing the testing process within a fast-paced CI/CD environment requires strategic operational thinking.

Dealing with Slow Tests

Slow tests are a common bottleneck, particularly E2E or complex integration tests. If tests become too slow, they delay feedback and can hinder developer productivity. Instead of simply accepting the slowdown or, worse, disabling valuable tests, consider these strategies:

• Optimize: Can the test itself be made more efficient? Is the environment it runs in slow? Investing in faster testing infrastructure can have a significant return on investment (ROI), as developer time is valuable. The increased feature velocity enabled by CI/CD should ideally outweigh marginal increases in testing costs.
• Parallelize: Can tests be run concurrently across multiple agents or environments? Many CI platforms and test runners support parallel execution.
• Categorize and Schedule: Separate tests into suites based on speed ("fast," "medium," "slow"). Run fast tests frequently (locally, on PRs), medium tests post-merge, and slow tests less often (e.g., nightly or pre-deployment).
• Prioritize: If you absolutely must reduce test execution time for a specific stage, prioritize running the tests covering the most critical functionalities or highest-risk areas first. Consider randomized sampling of less critical tests if full execution isn't feasible in the available time window.
• Re-evaluate Level: Is a slow E2E test verifying something that could be checked more quickly and reliably with a lower-level integration or unit test?

Prioritizing Bugs and Test Failures

Not all test failures or bugs have the same severity. When a test fails in the CI pipeline:

• Triage Immediately: Someone needs to quickly assess the failure. Is it a genuine bug in the code? A problem with the test itself (flaky)? An environment issue?
• Impact Assessment: How critical is the failure? Does it block a core user journey? Is it an edge case? This assessment informs the priority of fixing it.
• Don't Ignore Flaky Tests: While a flaky test might not represent a real regression this time, it erodes trust in the test suite. It needs to be investigated and fixed or quarantined (see Anti-patterns section).
• Production Failures: Failures detected in post-deployment tests running against production require immediate attention. The goal should be to quickly revert the deployment or apply a hotfix. Ensure your deployment process allows for easy and fast rollbacks.

Sometimes, especially in early product stages or when exploring new features, it might be acceptable to release with known, non-critical bugs. The strategy might involve releasing faster to gather user feedback, potentially using a beta program where engaged users actively look for issues in exchange for early access. However, this depends heavily on the product domain, user expectations, and risk tolerance.

The Role of QA

In a mature CI/CD environment, the role of dedicated QA professionals often shifts. With developers writing more automated tests (unit, integration) and the pipeline handling regression checks, QA can focus on higher-value activities that are difficult or impossible to automate:

• Exploratory Testing: Probing the application creatively to find unexpected issues.
• Usability Testing: Assessing the user experience.
• Complex Scenarios: Testing intricate workflows or edge cases not easily covered by automated scripts.
• Test Strategy & Planning: Helping define what needs testing and how best to achieve coverage.
• Analyzing Results: Interpreting trends in test failures and bug reports to identify systemic quality issues.
• Tooling & Automation Support: Helping select, implement, and maintain testing tools and frameworks.

QA should not be a bottleneck. Integrating testers early in the development process, fostering collaboration between developers and testers ("shift-left" testing), and ensuring clear responsibilities can streamline the quality assurance process. If manual testing processes consistently slow down releases, investigate which parts can be automated and ensure QA focuses on tasks requiring human insight. In some complex domains, outsourcing specialized testing (like security penetration testing or large-scale performance testing) might be considered.

Architectural Considerations

If bugs frequently emerge despite testing, or if tests are consistently difficult to write or maintain, it might indicate underlying architectural problems. Consider periodic architectural reviews to identify areas causing friction for testability or introducing excessive coupling.

Building and Managing Maintainable Tests

Tests are code, and they require the same care in design and maintenance as production code.

• Clarity and Readability: Use clear naming conventions (for tests and variables). Follow patterns like Arrange-Act-Assert (AAA) to structure tests logically. Add comments where necessary to explain complex setups or non-obvious assertions. Remember, others (or your future self) will need to understand and maintain these tests.
• Independence: Tests should ideally be independent of each other. One test's failure should not prevent others from running, nor should its execution leave behind state that affects subsequent tests. Use proper setup (TestInitialize, beforeEach) and teardown (TestCleanup, afterEach) mechanisms provided by your framework to manage state.
• Deterministic Behavior: Tests should produce the same result every time they are run against the same code, assuming no external factors change. Avoid dependencies on things like current date/time, random numbers (unless explicitly testing randomness and using fixed seeds), or uncontrolled external services within core functional tests. Use mocks and stubs to control dependencies.
• Focus: Each test should ideally verify a single logical concept or scenario. Tests trying to do too much become hard to debug when they fail.
• Abstraction and Patterns: For complex setup or repeated actions (like logging in for E2E tests), use helper functions or Page Object Models (in UI testing) to abstract details and reduce duplication. Create declarative tests where the intent is clear, hiding imperative setup details.
• Dependency Management: Avoid brittle dependencies. In infrastructure or environment setup, use version pinning (e.g., package-lock.json in Node.js, specific Docker image tags) rather than always pulling "latest," which can introduce unexpected changes.
• Test Impact Analysis: Understand how changes in production code might affect tests. Tools can sometimes help, but good organization (e.g., locating tests near the code they test) also aids developers in identifying potentially impacted tests manually.
• Equivalence Relations: When asserting equality, consider what level of equality matters. Does the order of elements in a list matter? Does floating-point precision need to be exact, or within a tolerance? Define assertions clearly. Sometimes, hash functions can serve as approximate equality checks for complex objects, though with potential for collisions.
• Retiring Tests: Tests aren't sacred. Regularly review your test suite. Tests that are consistently flaky despite fixing efforts, tests for removed features, or tests that are completely redundant due to newer, better tests should be considered for retirement. Deleting or rewriting a test requires as much consideration as creating one.

Correlating Failures and Root Cause Analysis (RCA)

When a bug slips through to production, or a test fails unexpectedly, effective analysis is key to improving the process.

• Bug Correlation: When a production bug is found, investigate: Was there a test that should have caught this? If yes, why didn't it (e.g., bug in test logic, incorrect assertion, flaky execution)? If no, write a new test (typically a regression test) that reproduces the bug before fixing it.
• Failure Tracking: Use CI/CD dashboards and test reporting tools to track failure history. Link test failures back to specific commits or changes (tools like git bisect can help identify when a regression was introduced).
• Root Cause Analysis: Don't just fix the symptom. Understand why the bug occurred or why the test failed. Was it a misunderstanding of requirements? A concurrency issue? An environmental difference? A faulty assumption in a mock? Addressing the root cause prevents similar issues in the future.

Handling Specific Challenges

Race Conditions and Asynchronous Processing

Testing code involving concurrency or asynchronous operations is notoriously tricky. Flakiness often arises here.

• Shared State: Be extremely careful when tests modify shared resources (static variables, shared files, database entries). Ensure proper cleanup or use techniques to isolate test runs (e.g., unique database names per run, transactions that get rolled back).
• Asynchronous Waits: If testing code that performs background work, don't rely on fixed delays (sleep(500ms)). This is unreliable. Use mechanisms provided by your language or framework, such as:
  • Callbacks or Promises/Futures/Async-Await to wait for completion.
  • Polling: Repeatedly check for an expected state change, with a reasonable timeout to prevent infinite loops if the condition is never met. Libraries often provide utilities for this ("wait for condition").
• Resource Contention: Ensure tests don't collide over limited resources like network ports. Use mechanisms to acquire resources exclusively or use dynamically assigned resources.
• Temporary Files/Folders: Use library functions designed to create unique temporary files or directories and ensure they are cleaned up afterward.
• Database Transactions: Where possible, wrap test database operations in transactions that are rolled back after the test, leaving the database in its original state.

Fuzzing

Fuzz testing (fuzzing) involves feeding unexpected, invalid, or random data into an application to see if it crashes or behaves unexpectedly. While often used in security testing, the principle can apply more broadly.

• Edge Cases: Ensure code handles minimum/maximum values, empty inputs, and unusually long inputs gracefully.
• Character Encodings: Be cautious when generating random strings; invalid UTF-8 sequences can cause issues in unexpected places.
• HTTP Timeouts: When fuzzing APIs, ensure client settings allow for potentially long-running calls if the fuzzer generates complex requests.

Maintaining a Consistent Environment

Differences between developer machines, CI runners, staging, and production are a major source of "it works on my machine" problems and test flakiness.

• Infrastructure as Code (IaC): Define environments using tools like Docker, Terraform, or Ansible to ensure consistency.
• Dependency Pinning: Lock down versions of OS packages, libraries, and tools (as mentioned before).
• Clean Slate: Ensure CI jobs start from a known clean state, deleting artifacts from previous runs.
• Configuration Management: Manage configuration differences between environments explicitly and carefully. Avoid hardcoding environment-specific values.
• Permissions: Ensure tests run with appropriate permissions (e.g., file system access) that match the target environment where possible, or mock interactions requiring special privileges if necessary.
• Canary Pipelines: For infrastructure changes (like updating the base OS image for CI runners or deployments), use a canary approach: route a small amount of traffic/builds to the new version first, monitor closely, and roll out more broadly only when confident.

Okay, let's continue with the chapter, focusing on common pitfalls like flaky tests and the crucial task of developing a robust testing strategy.

Anti-patterns in Testing

While tests are essential, certain common practices, or "anti-patterns," can undermine their value, waste effort, and even introduce instability.

The Bane of Flaky Tests

Perhaps the most frustrating anti-pattern is the flaky test. These are tests that produce inconsistent results – sometimes passing, sometimes failing – when run against the exact same code without any relevant changes.

• Why are they bad? Flaky tests destroy trust. When a test fails, developers should have confidence that it indicates a genuine problem. If tests fail randomly, developers start ignoring them ("Oh, that's just the flaky login test again"), builds get manually overridden, and real regressions can slip through unnoticed. They inject noise into the feedback loop, masking the real signal. A test that fails unpredictably provides very little reliable information.
• Why do they occur? Flakiness often stems from:
  • Race Conditions/Concurrency: Issues with timing in asynchronous operations or contention for shared resources (databases, ports, files).
  • Environment Differences: Subtle variations between test environments (local vs. CI, different CI agents).
  • Order Dependency: Tests that implicitly rely on other tests running first (or not running) to set up or clean up state.
  • Uncontrolled External Dependencies: Reliance on third-party services (APIs, networks) that might be slow, unavailable, or return varying data.
  • Infrastructure Issues: Intermittent network glitches, insufficient resources on test runners.
  • Non-Deterministic Code: Relying on factors like current time/date or unseeded random number generators within the test logic or the code under test.
  • Brittle Locators (UI Tests): Relying on unstable element IDs or CSS paths that change frequently.
  • Incorrect Timeouts/Waits: Insufficient waiting times for asynchronous operations to complete, especially under varying load conditions.
  • Resource Leaks: Tests not properly cleaning up resources (files, database entries, ports), causing conflicts for subsequent tests.
• Handling and Mitigating Flaky Tests:
  1. Prioritize Fixing: Treat flaky tests as high-priority bugs. Don't let them linger.
  2. Identify Them: CI platforms or test reporting tools can often help identify flaky tests by tracking pass/fail rates over time or supporting automatic reruns on failure. Running tests multiple times locally, potentially under stress (e.g., using tools like stress-ng on Linux to simulate load, or running tests in parallel), can sometimes reveal flakiness.
  3. Isolate and Debug: Reproduce the flakiness consistently if possible. Debug the test and the code it covers, looking for common causes like timing issues or resource conflicts.
  4. Improve Test Logic: Make assertions more robust, use reliable waiting mechanisms instead of fixed sleeps, ensure proper isolation and cleanup.
  5. Quarantine (Temporary): If a fix isn't immediate but the flakiness is blocking others, temporarily quarantine the test. This means marking it so it still runs but its failure doesn't fail the entire build. This should be a temporary measure, tracked with a high-priority bug ticket to fix it properly. Don't let the quarantine list grow indefinitely.
  6. Annotate: Some frameworks allow annotating tests as potentially flaky, perhaps triggering automatic retries within the CI pipeline. This can be a pragmatic step but doesn't fix the root cause.
  7. Consider Deletion: If a test is chronically flaky, difficult to fix, and its value is questionable or covered by other, more reliable tests, consider deleting it.

Remember, a UI flicker causing a test to fail might sometimes indicate a genuine usability issue, not just a test problem. Address the root cause, which might be in the application code itself.

Other Common Anti-patterns

• Testing on Production Resources (Q7): While testing in a production-like environment (Q1) is crucial, using actual production resources (especially databases with live customer data) for destructive or high-load testing is extremely dangerous and should generally be avoided. Data corruption or service disruption can occur. Use dedicated test accounts in production for smoke tests if necessary, or rely on high-fidelity staging environments.
• Lack of Production-Like Environment (Q1): The inverse problem. If the test environment doesn't closely mirror production (configuration, data characteristics, infrastructure), tests might pass but miss issues that only manifest in the real world. Strive to keep staging/PPE environments as close to production as possible, using IaC and configuration management.
• Blindly Chasing Coverage Thresholds (Q4): As discussed earlier, focusing solely on hitting a coverage percentage leads to low-value tests. Using previous builds' coverage as a fixed target (Q3) is also problematic, as removing well-tested legacy code could artificially lower coverage, penalizing necessary cleanup.
• Manual Execution of Automated Checks (Q8): If tests are designed for automation (deterministic inputs, clear pass/fail criteria) but are still executed manually, it negates the speed and consistency benefits of CI/CD. Automate what can be reliably automated.
• Ignoring Test Maintenance: Treating tests as write-once artifacts. Tests need refactoring, updating, and retiring just like production code.

Automated vs. Manual Testing: A Necessary Partnership

It's common to hear debates about "automated vs. manual" testing, often positioning them as opposing forces. However, as Michael Bolton and others argue, this is largely a false dichotomy. They are different activities with different strengths, and a mature testing strategy needs both.

• Automated Checks: What we typically call "automated testing" is more accurately described as automated checking. Computers excel at executing predefined steps and verifying expected outcomes against specific, unambiguous criteria. They are fast, consistent, tireless, and ideal for regression checking, verifying known invariants, and covering many scenarios quickly. They handle the repetitive verification that humans are ill-suited for.
• Human-Centric Testing: "Manual testing" should not mean humans manually executing automatable scripts. Instead, it leverages unique human capabilities:
  • Exploration & Learning: Exploring the application, learning how it works, identifying usability issues, questioning assumptions, and finding unexpected bugs that no script was designed to look for. This is exploratory testing.
  • Subjectivity & Experience: Assessing qualities like usability, aesthetics, clarity, and overall user experience – things computers struggle to quantify.
  • Tacit Knowledge: Applying intuition and experience built from understanding users, the domain, and past issues.
  • Adaptability: Designing and modifying tests on the fly based on observations.
  • Critical Thinking: Evaluating if the software meets the intent behind the requirements, not just the letter of the specification.

Computers check conformance to specifications; humans evaluate fitness for purpose. Relying solely on automated checks leaves blind spots regarding usability, discoverability, and unexpected interactions. Relying solely on manual effort for things computers can check reliably is inefficient and slow.

In CI/CD, automated checks are essential for the fast feedback loop and regression safety net. Human-centric testing complements this by providing deeper insights, evaluating user experience, and finding bugs that automation misses. The goal is to automate the checks to free up human testers to focus on testing (evaluation, exploration, learning).

Developing a Test Strategy

Given that you can't test everything, and different tests serve different purposes, you need a test strategy. This is a plan outlining the approach to testing for a specific project or product. It defines what to test, how to test it (which types of tests, tools), when to test (at which pipeline stages), and who is responsible, all aligned with business goals, risk tolerance, and available resources.

Why Do You Need a Strategy?

• Finite Resources: Time, budget, and people are limited. A strategy helps allocate these resources effectively to maximize value and mitigate the most significant risks.
• Complexity: Modern applications are complex. A strategy provides a framework for tackling this complexity systematically.
• Alignment: Ensures the testing effort supports business objectives (e.g., rapid feature delivery vs. extremely high reliability).
• Consistency: Provides a common approach for the team.

Key Questions to Address:

• What are the goals? What does "quality" mean for this product? What are the critical user journeys? What are the biggest risks (technical, business, security)?
• What is the risk appetite? Is this a life-critical system where bugs are unacceptable, or a fast-moving consumer app where some imperfections might be tolerated in exchange for speed?
• What types of tests are needed? Based on the application architecture and risks, what mix of unit, integration, E2E, performance, security, and manual exploratory testing is appropriate?
• How will tests be implemented and managed? Which frameworks and tools? How will tests be organized and maintained?
• When will tests run? Define the testing stages within the CI/CD pipeline.
• How will results be analyzed and acted upon? Define the process for handling failures, tracking metrics, and improving the strategy over time.

Balancing Quality, Speed, and Cost

Testing exists within the classic project management triangle:

• Quality: How reliable, usable, and performant the software is. More testing generally aims for higher quality.
• Speed: How quickly features can be delivered to users. Extensive testing can slow down delivery cycles.
• Cost: The resources (people, infrastructure, tools) required for testing.

A test strategy must find the right balance based on context. A startup prioritizing market fit might lean towards speed, accepting slightly lower initial quality (and relying more on user feedback and fast iteration), while a financial institution might prioritize quality and regulatory compliance, accepting higher costs and slower delivery. There's no single "right" balance; it's context-dependent.

Risk-Based Testing (RBT)

A common approach to prioritize testing efforts is Risk-Based Testing. This involves identifying areas of the application with the highest risk (likelihood of failure * impact of failure) and focusing testing resources there.

• Identify Risks: Brainstorm potential problems. Consider:
  • Complex features
  • Frequently changed areas
  • Business-critical functionalities (e.g., payment processing)
  • Integration points with external systems
  • Security-sensitive areas
  • Areas with a history of bugs
  • Performance-sensitive operations
• Assess Likelihood and Impact: Estimate how likely each risk is to occur and how severe the consequences would be if it did.
• Prioritize: Focus testing effort on high-risk items first. Low-risk items might receive less intensive testing or rely more on basic smoke tests.

Caveats of RBT:

• Subjectivity: Risk assessment is inherently subjective and can be biased. Involving multiple stakeholders helps.
• Blind Spots: Focusing only on known high risks might neglect testing newer or less understood areas where "unknown unknowns" might lurk. It can also de-prioritize non-functional requirements like usability or long-term maintainability if they aren't framed as immediate risks.
• The Long Tail: While focusing on the top risks is efficient initially, neglecting the "long tail" of lower-risk items entirely can lead to an accumulation of minor issues that eventually impact quality or user experience.
• Diminishing Returns: After addressing major risks, finely prioritizing among many small, similar risks can become difficult and bureaucratic.

RBT is a valuable tool for initial prioritization but shouldn't be the only factor. Combine it with coverage goals for critical areas and dedicated time for exploratory testing to mitigate its potential blind spots. Use risk to guide the intensity and order of testing, but ensure a baseline level of testing exists even for lower-risk areas.

Other Prioritization Factors

Beyond pure risk, consider:

• Usage Data: Prioritize testing frequently used features (based on analytics).
• Customer Impact: Focus on areas impacting high-value customers or core workflows.
• Regulatory Requirements: Mandated testing for compliance (e.g., accessibility, data privacy).
• Team Expertise: Leverage team members' knowledge of historically problematic areas.

Should I Write a Test For It? The Pragmatic Approach

When faced with a specific piece of code or functionality, ask:

• Is the behavior critical or complex? If yes, it likely warrants a dedicated test.
• Is it likely to break due to future changes? Tests act as future-proofing.
• Can it be verified effectively at a lower level? Prefer unit/integration tests over E2E if they provide sufficient confidence faster.
• Is it already covered adequately by other tests (manual or automated)? Avoid redundant effort.
• Is the behavior easily demonstrable and verifiable? If the expected outcome is clear and stable, it's a good candidate for an automated check. If it's highly subjective or rapidly changing (like early UI prototypes), extensive automated tests might be premature.
• What's the cost/benefit? How long will the test take to write and maintain vs. the risk of not having it?

Be pragmatic. In a fast-moving startup with evolving requirements, writing comprehensive E2E tests for every minor UI tweak might be counterproductive. Focus initial automated tests on core logic and critical paths. In a mature, stable application, more extensive regression testing is appropriate. Adapt your strategy to the project's lifecycle stage and risk profile. Look at past bugs – they are excellent indicators of where your previous testing strategy might have had gaps.

When Should a Test Fail? Finding the Right Sensitivity

Tests check for deviations from expectations. But how much deviation should trigger a failure?

• Exact Match: For calculations or specific data outputs, an exact match might be required.
• Thresholds: For performance tests or floating-point comparisons, failing only if a value exceeds a certain threshold or differs by more than a small epsilon might be appropriate.
• UI Brittleness: UI tests are prone to this. Should a test fail if a button's color changes slightly? If it moves 2 pixels? If its internal ID changes but its text remains the same? Relying on volatile implementation details (like exact CSS paths or generated IDs) makes tests brittle. Prefer testing based on user-visible attributes (text content, accessibility roles, dedicated data-testid attributes) where possible.
• Snapshot Testing: Tools can capture a "snapshot" (e.g., of a UI component's rendered output or an API response structure) and fail the test if the snapshot changes. This catches unexpected changes but requires manual review and updating of the snapshot whenever an intentional change occurs. It can be useful but requires discipline.

The goal is to make tests fail when meaningful changes occur but remain resilient to irrelevant implementation details. This often involves careful selection of assertion methods and UI locators. Allow manual overrides for test failures in CI pipelines, but only with scrutiny – is the failure truly insignificant, or is it masking a real issue?

Okay, let's wrap up the chapter on testing, bringing together the strategies and philosophies discussed.

Refining Your Strategy: Choosing the Right Tests

We've established that blindly following the testing pyramid isn't optimal. The core principle remains: test at the appropriate level to gain the necessary confidence efficiently. How do you decide between unit, integration, or E2E?

• Too many isolated unit tests: Can lead to a situation where individual components work perfectly alone, but the integrated whole fails. You might have 100% unit test coverage, but event handlers aren't connected, data doesn't flow correctly between services, or buttons simply don't trigger the right actions in the complete application.
• Over-reliance on mocked dependencies: Mocking is essential for unit testing, but tests relying heavily on mocks provide less confidence about real-world interactions. If your tests mock all external services, you aren't verifying the actual contracts or handling real network latency/errors. At some point, you need integration tests that interact with real (or realistic, like containerized) dependencies. If an external service is genuinely flaky in production, your integration tests should reflect that (perhaps with retry logic mirroring production) to provide realistic feedback. If it's slow in production, your tests reflecting that slowness provide valuable performance insights, though you need to balance this with feedback loop time.
• When implementation details matter (Unit/Integration): Consider the cache example again. If the process of retrieving data (i.e., hitting the cache vs. the database) is what you need to verify, an E2E test checking only the final data is insufficient. You need a lower-level test that can inspect or control the internal behavior (e.g., mocking the DB and asserting it wasn't called). Similarly, verifying internal state changes or calls to private/internal methods (like audit logging) often requires unit or integration tests.
• When the integrated outcome matters (E2E): If you need to verify a user can complete a multi-step workflow across different parts of the UI and backend services, an E2E test is often the most direct approach. Testing if a button is visible and clickable within the context of the entire application page requires an E2E perspective; a unit test of the button component in isolation doesn't guarantee it renders correctly or is accessible in the final assembly.

Think about the opposite situation: How would you know if this didn't work? What's the simplest, fastest test that could reliably detect that failure? Often, this mental model helps choose the right test level.

Happy Path vs. Sad Path Testing

• Happy Path: This tests the ideal, error-free scenario where the user does everything correctly (provides valid input, follows the expected sequence). Example: Successfully logging in with correct credentials, adding an item to the cart, and checking out smoothly. Happy path tests are essential to verify core functionality works under normal conditions.
• Sad Path: This tests scenarios involving errors, invalid input, or unexpected user actions. Example: Trying to log in with an incorrect password, attempting to add an expired coupon code, submitting a form with missing required fields, transferring a negative amount of money. Sad path tests are crucial for ensuring the application handles errors gracefully and provides informative feedback rather than crashing or producing incorrect results.

A balanced test strategy needs both. Over-focusing on the happy path leaves the application vulnerable to breaking under common error conditions or user mistakes. Over-focusing only on edge cases and errors might mean the core, successful workflows aren't adequately verified. Aim to cover the main happy paths and the most probable or impactful sad paths. Techniques like fuzzing can help explore less obvious sad paths.

Mutation Testing Revisited

As mentioned earlier, mutation testing provides a stricter assessment of test suite quality than code coverage. By making small code changes and checking if your tests fail ("kill the mutant"), it verifies if your tests are sensitive enough to detect actual code alterations. While computationally intensive, incorporating periodic mutation testing runs (perhaps less frequently than standard tests) can provide deeper confidence in your test suite's effectiveness, especially for critical logic. It helps counteract the weakness of tests that achieve high code coverage but lack meaningful assertions.

Retiring Tests: When to Let Go

Tests incur a maintenance cost. Just as features become obsolete, so can tests. Consider retiring or significantly rewriting tests when:

• The Feature is Removed: If the functionality a test covers is deleted from the application, the test is no longer needed.
• Redundancy: A newer, better test (perhaps at a different level, like an integration test covering what several brittle unit tests did) now provides the same or better coverage more reliably.
• Chronic Flakiness: If a test is persistently flaky despite significant effort to stabilize it, and its value doesn't justify the ongoing disruption and maintenance burden, deletion might be the best option (assuming the coverage is acceptable or replaced).
• Low Value & High Maintenance: If a test covers a very low-risk, stable area of the code but is complex and frequently breaks due to unrelated refactoring, its maintenance cost might outweigh its benefit.

Retiring tests should be done thoughtfully. Ensure you understand what coverage is being lost and that it's acceptable given the current risk assessment and overall test strategy.

Testing: The Safety Net for CI/CD

Continuous Integration and Continuous Deployment are built on the principle of making small, frequent changes and getting them to production quickly and reliably. Automated testing is the essential safety net that makes this possible.

Without a robust testing strategy integrated into the pipeline:

• CI becomes Risky Integration: Merging code frequently without validation leads to an unstable main branch ("integration hell").
• CD becomes Continuous Disaster: Deploying unverified changes frequently leads to production failures, eroding user trust and requiring constant firefighting.

Testing provides the confidence needed to automate the integration and deployment process. It enables:

• Fast Feedback: Catching errors early in the cycle, reducing the cost of fixing them.
• Risk Reduction: Decreasing the likelihood of deploying breaking changes to production.
• Increased Velocity: Allowing developers to merge and deploy more frequently and confidently, knowing the safety net is in place.
• Improved Code Quality: Encouraging testable design and providing a regression suite to prevent degradation over time.
• Collaboration: Providing a shared understanding of expected behavior and system health.

Conclusion

Testing in a CI/CD world is not just about finding bugs; it's a fundamental pillar supporting the entire philosophy of rapid, reliable software delivery. It requires a strategic approach, balancing different types of tests, managing them effectively, and understanding their strengths and limitations. From fast unit tests providing immediate feedback to developers, through integration tests verifying component interactions, to E2E tests validating user journeys, and including vital human-centric exploratory and usability testing, a comprehensive strategy is key.

By embracing testing not as a separate phase but as an integral part of the development workflow, embedded within the CI/CD pipeline, teams can gain the confidence needed to innovate faster, deploy more frequently, and ultimately deliver higher-quality software that truly meets user needs. Remember, tests are only as good as the effort and strategy behind them – invest wisely, and they will pay dividends in stability, speed, and confidence.-

Okay, let's start building the "Dependency Management" chapter for your "Beginning CI/CD" book based on your notes. I'll structure it logically, expand on the concepts, integrate the analogies, and aim for a clear, beginner-friendly tone suitable for your audience.

Here is the first part of the chapter:

Chapter X: Dependency Management - The Unsung Hero of CI/CD

Introduction to Dependency Management

Imagine you're setting up your Continuous Integration and Continuous Delivery (CI/CD) pipeline. You want automated builds, tests, and deployments humming along smoothly. But what happens when your build fails because a specific library wasn't found, or your tests pass in your local environment but fail in CI because of a version mismatch? These common frustrations often stem from the world of dependency management.

In the context of CI/CD, robust dependency management isn't just a "nice-to-have"; it's fundamental. It ensures reproducibility, meaning your software builds consistently everywhere – on your machine, your colleague's machine, and crucially, in your CI pipeline. It guarantees stability by controlling the exact versions of external code your application relies on. And it enhances security by making it easier to track and update components with known vulnerabilities. Mastering dependency management is a key step towards achieving reliable and efficient CI/CD.

What are Dependencies?

At its core, a dependency is an external piece of software – like a library, framework, or package – that your application requires to function correctly. Think of them as pre-built components or tools that save you from reinventing the wheel.

Let's use a simple analogy: baking a cake. To bake your delicious chocolate cake, you need ingredients like flour, sugar, cocoa powder, eggs, and milk. You don't need to grow the wheat and mill the flour yourself, nor do you need to raise chickens for eggs or keep a cow for milk. These ingredients are your dependencies. You rely on them being available and correct for your cake recipe (your application) to succeed. If the grocery store gives you salt instead of sugar, your cake won't turn out right. Similarly, if your application expects version 1.0 of a library but gets version 2.0 with breaking changes, it will likely fail.

A Brief History: Why We Need Dependency Management

Managing dependencies wasn't always the complex task it can seem today. Let's take a quick journey through time to see how the need evolved:

• 1950s-1960s (Assembly & Early High-Level Languages): Software was often custom-built for specific hardware. Sharing code was rare, and dependencies were minimal or manually handled. The advent of languages like FORTRAN and COBOL, along with early operating systems like UNIX introducing shared libraries, planted the seeds of code reuse, but formal management was nonexistent.
• 1970s-1980s (Linkers & Early Version Control): Tools like linkers emerged to combine different code pieces (object files, libraries) into a runnable program – an early form of dependency resolution. Version control systems like SCCS and RCS appeared, helping track changes to code, which indirectly aided in managing different versions of software components.
• 1990s (Build Tools & OS Package Managers): Build automation tools like make became common, helping manage the compilation dependencies. On the operating system level, package managers like RPM (Red Hat) and dpkg (Debian) arrived to manage software installation and dependencies for the entire system. They prevented system-level conflicts but didn't solve issues within specific application projects.
• 2000s (Language-Specific Managers & SemVer): This was the breakthrough era. Tools tailored for specific programming languages exploded: Maven (Java), npm (JavaScript), pip (Python), Bundler (Ruby), NuGet (.NET), and many others. They focused on managing dependencies for a single project, often isolating them from other projects. The concept of Semantic Versioning (SemVer) was introduced, providing a standardized way to communicate the impact of version changes (more on this later!).
• 2010s-Present (Containers, Microservices & Security Focus): Containerization technologies like Docker took isolation to the next level, packaging an application and all its dependencies together. The rise of microservices introduced the challenge of managing dependencies between services. Furthermore, awareness of software supply chain security grew dramatically, leading to tools and practices focused on scanning dependencies for known vulnerabilities.

This evolution highlights a clear trend: as software complexity and the practice of code reuse grew, the need for automated, reliable, and sophisticated dependency management became paramount. Manual management simply doesn't scale and introduces too many risks.

The Role of Package Managers

So, how do we solve the problems of manual dependency wrangling? Enter the package manager. You might initially see it as just another tool to learn, perhaps even bureaucratic overhead. However, package managers are essential assistants designed to streamline the complex task of handling dependencies.

Think back to our baking analogy. Instead of going to the grocery store yourself, listing every ingredient, checking your pantry for duplicates, and carrying everything home, imagine you have an assistant. You just give the assistant your recipes (for the chocolate cake and maybe some cookies that share some ingredients like flour and sugar). The assistant:

1. Figures out the total list: They see both recipes need flour, sugar, eggs, etc., but cookies also need chocolate chips.
2. Checks your pantry (your project): They see you already have plenty of flour.
3. Goes to the store (package repository): They know exactly where to find reliable ingredients (standardized packages).
4. Gets exactly what's needed: They buy sugar, eggs, cocoa powder, milk, butter, vanilla, and chocolate chips – without buying extra flour you already have.
5. Ensures compatibility: They make sure to get baking soda and baking powder if both are required, not substituting one for the other incorrectly.
6. Stocks your pantry correctly: They put the ingredients away neatly (install packages in your project, often in a specific folder like node_modules or vendor).