Search for a command to run...
Simplifying CI/CD on Azure DevOps: Understanding Pipeline as Code, Azure Repos & Classic Editor
Passionate about helping organizations build scalable infrastructure and DevOps solutions with cloud technologies. Experienced in designing robust systems, automating processes, and driving efficiency through innovative cloud solutions. Advocate for best practices in DevOps and cloud computing, committed to enabling teams to achieve their full potential.
No comments yet. Be the first to comment.
From StorageClass to LoadBalancer, building a fully functional MySQL-backed web application on AKS.
Learn to Use Python for OCR, Object Detection, Tagging, and Captioning with Azure AI Vision
Using Retrieval-Augmented Generation with LLM in Azure
Covers interview questions on Git, Docker, Kubernetes, Terraform, Ansible, Azure DevOps, and Azure Cloud
Efficient Microservices Deployment with Azure DevOps CI & GitOps with ArgoCD to AKS
Azure DevOps is a suite of development tools Microsoft provides to support the complete software development lifecycle. It encompasses various tools for planning, developing, delivering, and maintaining software projects, enabling teams to collaborate more effectively and deliver high-quality software faster.
Why Azure DevOps
Services Provided by Azure DevOps
Getting Started with Azure DevOps
Azure Repos
Azure DevOps Build Pipeline
Classic Editor Pipeline
YAML Pipeline
Self-Hosted Agent
All in One: Azure DevOps provides a complete suite of tools for the entire software development lifecycle, including source control, CI/CD pipelines, project management, testing, and artifact management.
Seamless Integration: All the tools are seamlessly integrated, reducing the need to switch between different tools and ensuring a smooth workflow.
Third-Party Integration: With a vast extensions marketplace, Azure DevOps can integrate with many third-party tools and services, enhancing its functionality and adaptability.
All-in-one Place: A single platform for development, project management, and operations enhances team collaboration and reduces communication gaps.
Multi-Cloud Deployments: While optimized for Azure, it supports deployments to other cloud providers like AWS and Google Cloud.
Azure Repos: Git repositories for source control and version management.
Azure Pipelines: CI/CD pipelines for automated build, test, and deployment.
Azure Boards: Agile tools for planning, tracking, and project management.
Azure Artifacts: Package management for Maven, npm, NuGet, and more.
Azure Test Plans: Tools for manual and automated testing.
Before setting up your Azure devops, you need to have an account with Microsoft, and with this article, I assume you already have an account, so let's move on to the Azure DevOps: Azure DevOps and sign in with your MS account.
After signing up you need to create an Organization, an organization is nothing but a top-level fundamental container that serves as a namespace for managing and organizing your DevOps resources. The organization also facilitates collaboration by allowing multiple teams to work on different projects within the same organizational framework.
Once a captcha authentication is completed, you'll be able to name your organization and create it which further lands you on the project creation page where you can give a name for your project and get started:
Now, the project has been created under the organization called Hashnode Demo
Within an organization, you can create multiple projects. Each project is a container for source code, builds, releases, test plans, and other resources.
Azure Repos is nothing but a set of version control tools provided by Microsoft as part of the Azure DevOps suite which supports both Git repositories and Team Foundation Version Control(TFVC) to manage our code base. Azure repo is designed to support any size of the team and enterprise application. The TFVC is now obsoleted and Git-based repos have been used widely across the industries.
Fully distributed Version control system.
Supports branching, merging, pull requests, and more.
Ideal for teams using modern version control workflows.
Centralized version control system.
Suitable for teams that prefer a more traditional version control approach.
Supports large codebases and binary files.
Pull Requests:
Facilitates code reviews by enabling team members to comment on and review code changes.
Allows for automated builds and tests to be triggered as part of the pull request process.
Branch Policies:
Enforce policies on branches to ensure code quality and compliance.
Require pull request reviews, successful builds, and code coverage before merging.
Code Search:
Powerful search capabilities to find code across repositories.
Helps in quickly locating definitions, references, and changes.
Web-Based Code Editing:
Edit code directly in the browser without needing a local development environment.
Useful for quick changes and fixes.
Integration with CI/CD Pipelines:
Seamlessly integrates with Azure Pipelines for continuous integration and continuous deployment.
Automate builds, tests, and deployments based on repository changes.
Support for Git Hooks:
Implement custom scripts to run at different points in the Git workflow.
Automate and enforce development workflows and practices.
Setting up Azure Repos involves creating a new project, initializing a repository, and configuring your development environment to work with it. It also allows you to integrate your existing repositories from Git-Hub or similar platforms. In this article I will be demonstrating 2 ways:
Cloning a new repository to your computer from Azure Repos is a simple process. It allows you to create an empty repository, which you can start using immediately without the need for explicit initialization. This process is similar to cloning a repository from GitHub, and it provides a straightforward way to set up your local development environment.
Simply copy the given URL and run the below command
git clonehttps://Org-Hashnode@dev.azure.com/Org-Hashnode/Hashnode%20Demo/_git/Hashnode%20Demo
Once you run the command it will prompt you to authorize via your Microsoft account
Importing your existing repository from a different platform, such as GitHub, to Azure Repos is a seamless process. This is particularly useful when you want to migrate a large codebase and continue development within Azure DevOps.
Click on Import
This will further prompt you to choose the repository type out of Git or TFVC and paste your existing repository clone URL
Now you can copy your existing repository HTTPS clone URL and paste it in the Clone URL box:
Here we're using a YouTube Clone application based on NodeJS for demonstration purposes that has been forked from Adrianhajdin's repo.
Now, click on Import. This will migrate your repository to Azure Repos, eliminating the dependency on GitHub. By importing your repository, you ensure that all your code, branches, and commit history are now managed within Azure DevOps, providing a seamless and integrated development environment.
Azure Pipelines is a powerful and versatile service provided by Microsoft Azure DevOps. It enables you to build, test, and deploy your code automatically and continuously, ensuring high-quality software delivery. Azure Pipelines supports a wide range of languages, frameworks, and platforms, making it a comprehensive solution for continuous integration (CI) and continuous deployment (CD). Azure Pipelines provides the flexibility, scalability, and reliability needed to support modern DevOps practices.
Azure Pipelines can be created using different methods, each catering to various preferences and project requirements. There are two primary ways to create pipelines in Azure DevOps:
In this article, we'll be covering both ways of creating pipelines.
The Classic Editor in Azure DevOps Pipelines provides a graphical user interface (GUI) for creating and managing build and release pipelines. This method is particularly useful for users who prefer a visual approach or are new to pipeline configuration. The Classic Editor offers an intuitive way to define the steps and stages of your CI/CD process without writing YAML code.
Pre-Built Tasks:
A library of pre-built tasks and templates to quickly configure common build and deployment scenarios.
Pipeline Stages and Jobs:
Visual representation of stages and jobs, making it easier to understand and manage complex workflows.
Variable Management:
Easy management of pipeline variables and secrets through the GUI.
Trigger Configuration:
Simple setup for continuous integration (CI) and continuous delivery (CD) triggers.
Integrations:
Seamless integration with various source control systems, build agents, and deployment environments.
As we have already configured the Azure Repos now it's time to build and deploy the application using the classic editor which will help us understand the workflow better.
NB: By default, the creation of classic build and release pipelines is disabled in Azure DevOps. To enable the Classic Editor option at the project level, you need to make changes at the organization level as shown below
Make sure both these options are toggled off to enable the classic editor option at the project level. you will allow the use of the Classic Editor for both build and release pipelines at the project level. This setting ensures that you have access to the Classic Editor’s graphical interface for creating and managing your pipelines.
Now you can see that Use the classic editor option is enabled at the project level below
Now once click/ on the hyperlink it will land you on the page on which the source of your code base needs to be chosen
We have chosen Azure Repos Git as we have already imported our code to Azure Repos. You can also choose other options such as GitHub, Bitbucket Cloud, etc as displayed. The underlying functionality is the same as all of the source options. Click on continue.
Now you can select a template or proceed with an Empty job.
We'll proceed with the Empty job as we are going to configure our custom steps. Once clicked, you will be enabled to add Steps for your Job as shown below:
In this demonstration, the pipeline job is named "Build and Deploy App" and encompasses multiple steps. For this basic pipeline example, we handle both the build and release processes within a single pipeline.
The first step in creating any pipeline is configuring the agent where the pipeline jobs will be executed. An agent is essentially a server that performs the tasks required to build the application and generate the necessary artifacts.
Agent Definition: The agent is a crucial component of the pipeline infrastructure. It executes the tasks defined in the pipeline, such as building the code, running tests, and deploying artifacts.
Agent Pools: You can use either Microsoft-hosted agents or configure your own self-hosted agents. Microsoft-hosted agents are managed by Azure DevOps and come pre-installed with commonly used tools. Self-hosted agents are machines you set up and configure yourself, offering greater control and customization.
Selecting an Agent: During pipeline creation, you'll specify which agent or agent pool will be used to run the jobs. This selection determines the environment in which your pipeline tasks will be executed.
NB: As part of this demonstration, we'll be using self-hosted agents. Creating and configuring self-hosted agents will be demonstrated in later parts of this article.
To define tasks in Classic Editor you can take the help of existing templates by searching them in the search bar and choosing the required template that will be added to the left as shown in further steps
npm install
Once the template is chosen it will be added to the left and in the right section you can fill in the necessary fields such as command, working folder, variables, etc.. for that particular task as shown above.
npm build
The subsequent task is the build task where the application will be built by the command specified as run build. These are NodeJS-based command and not related to AZDO
Publish Artifacts
Once the application code is built, it generates the deployable artifacts that need to be published to the hosting server.
Deploy to App Service
As our artifacts are ready, now using the Azure App Service Deploy template, we'll be deploying the artifacts/deployable application to Azure App Service.
Now click on Queue or Save & queue to trigger the pipeline which will further ask you for the Agent pool and Branch/tag
As we have taken self-hosted agents, I have placed them in the default agent pool hence, Default has been chosen.
Once Run is clicked the pipeline execution will be queued
The execution will begin as below
As shown above, once the pipeline execution is completed, the application has been deployed to Azure App Service and we can access the application via the red-bordered URL as shown below
A YAML pipeline in Azure DevOps is defined using a YAML (YAML Ain't Markup Language) file. This file contains the pipeline configuration as code, making it easy to version, share, and reuse. YAML pipelines provide a way to automate the build, test, and deployment processes in a structured, text-based format.
Version Control:
Flexibility and Customization:
Infrastructure as Code:
Collaboration:
Click on the Create Pipeline on the project pipeline section which will land you on the below page asking you to choose where exactly your pipeline code resides
As we are using Azure Repos for this demonstration, the same needs to be selected.
Once source is selected it will as you to configure the pipeline using pipelin-code. Here we have multiple options that can assist us in creating the pipeline with templates, it also allows you to import pipeline code if it already exists.
In our case, we'll choose the Starter pipeline as we are going to create a pipeline from scratch so starter pipeline will provide a basic format to get started with like below
Its architecture is hierarchical, consisting of triggers, stages, jobs, steps, and tasks, each playing a distinct role in the CI/CD process.
Trigger
Triggers define when the pipeline should be run automatically. They can be configured based on various events, such as code commits, pull requests, or on a schedule.
Types:
CI (Continuous Integration) Triggers: Automatically run the pipeline when code is committed to a branch.
PR (Pull Request) Triggers: Run the pipeline when a pull request is created or updated.
Scheduled Triggers: Run the pipeline at specified times.
Resource Triggers: Run the pipeline based on changes in external resources like container images.
Example:
---
trigger:
- main
pool:
name: Default
Stage
Stages are the major phases of the pipeline, grouping jobs that logically belong together. Stages can run sequentially or in parallel.Features:
Isolation: Each stage can be executed in a different environment.
Dependencies: Stages can have dependencies, ensuring they run in a specific order.
stages:
- stage: Build
jobs:
- job: Build
pool:
name: Default
steps:
- task: Npm@1
displayName: NPM Install
inputs:
command: 'install'
verbose: true
- task: Npm@1
inputs:
command: 'custom'
customCommand: 'run build'
- job: Publish_Artifact
displayName: Publish Artifact
pool:
name: Default
steps:
- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: 'build'
ArtifactName: 'drop'
- stage: Deploy
jobs:
- job: Deploy
pool:
name: Default
steps:
- task: AzureRmWebAppDeployment@4
inputs:
ConnectionType: 'AzureRM'
azureSubscription: 'Pay-As-You-Go(4accce4f-9342-4b5d-bf6b-5456d8fa879d)'
appType: 'webAppLinux'
WebAppName: 'youtube-devopswithritesh'
packageForLinux: $(System.DefaultWorkingDirectory)/build
RuntimeStack: 'STATICSITE|1.0'
Job
Jobs are units of work that run on an agent. Each job consists of a series of steps and can run independently or in sequence with other jobs.
Features:
Parallelism: Multiple jobs can run in parallel.
Agent Specification: Jobs specify the agent or pool on which they should run.
Example:
- job: Build
pool:
name: Default
steps:
- task: Npm@1
displayName: NPM Install
inputs:
command: 'install'
verbose: true
- task: Npm@1
inputs:
command: 'custom'
customCommand: 'run build'
Step
Steps are the individual actions performed in a job. Each step represents a single task, such as running a script or executing a command.
Features:
Sequential Execution: Steps within a job run sequentially.
Conditionals: Steps can have conditions that determine their execution.
Example:
steps:
- task: Npm@1
displayName: NPM Install
inputs:
command: 'install'
verbose: true
- task: Npm@1
inputs:
command: 'custom'
customCommand: 'run build'
Task
Tasks are predefined actions or operations that can be executed within a step. Azure DevOps provides a library of built-in tasks, and you can also define custom tasks.
Types:
Built-in Tasks: Provided by Azure DevOps for common operations like building code, running tests, and deploying applications.
Custom Tasks: Custom scripts or actions defined by the user.
Example:
steps:
- task: Npm@1
displayName: NPM Install
inputs:
command: 'install'
verbose: true
- task: Npm@1
inputs:
command: 'custom'
customCommand: 'run build'
While creating pipeline code you can also take the help of pipeline assistance by clicking on Show Assistant which will assist you with pre-built templates for each task.
You can also customize the template by putting custom values
Once the pipeline is ready, you can run it by clicking on the Run button. If you are running the pipeline for the first time, you might need to authorize the pipeline to access the agent pool, especially if you are using self-hosted agents. This authorization ensures that the pipeline has the necessary permissions to utilize the resources provided by the agent pool.
Now the execution has been started and we have 2 stages that are Build and Deploy stages respectively.
In the above picture, you can see that the Build stage has been completed. However, it is not proceeding to the Deploy stage, causing the Deploy stage to remain in a pending state indefinitely. This issue occurs because artifacts generated in the Build stage are not automatically passed to the Deploy stage.
To resolve this, you need to ensure that the artifacts produced in the Build stage are available to the Deploy stage. This can be achieved by downloading the artifacts from the Build stage in the Deploy stage.
Now you can see that another task named DownloadBuildArtifacts has been added to the pipeline, which we have implemented with the help of the built-in assistance. The DownloadBuildArtifacts task is a template provided by Azure DevOps that simplifies the process of transferring artifacts between stages.
Using the DownloadBuildArtifacts task ensures that the artifacts generated in the Build stage are available in the Deploy stage, facilitating a smooth transition between stages.
- stage: Deploy
jobs:
- job: Deploy
pool:
name: Default
steps:
- task: DownloadBuildArtifacts@1
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)'
- task: AzureRmWebAppDeployment@4
inputs:
ConnectionType: 'AzureRM'
azureSubscription: 'Pay-As-You-Go(4accce4f-9342-4b5d-bf6b-5456d8fa879d)'
appType: 'webAppLinux'
WebAppName: 'youtube-devopswithritesh'
packageForLinux: '$(System.ArtifactsDirectory)/drop'
RuntimeStack: 'STATICSITE|1.0'
Now, after adding the DownloadBuildArtifacts task, the Deploy stage can access the artifacts generated in the Build stage. As a result, the deployment has been successfully completed.
By using the DownloadBuildArtifacts task, we ensured that the artifacts produced in the Build stage were properly transferred to the Deploy stage, enabling the deployment process to proceed without any interruptions.
Finally, the application has been deployed and we can access it.
A self-hosted agent is a machine that you manage to run your Azure DevOps pipeline jobs. Unlike Microsoft-hosted agents, which are managed by Azure DevOps, self-hosted agents provide greater control over the environment, tools, and resources used during the build and deployment process.
Prepare Your Machine:
Ensure your machine meets the prerequisites. It should have an operating system supported by Azure DevOps (Windows, Linux, macOS).
Install necessary dependencies (e.g., .NET Core, Node.js, Docker, etc.) that satisfy your project needs. As our application is based on NodeJS all the Node dependencies have been installed on my machine.
Here I am using an EC2 instance hosted in the AWS cloud.
Create an Agent Pool:
Navigate to your Azure DevOps organization.
Click on "Organization settings" (gear icon) in the lower-left corner.
Select "Agent pools" under "Pipelines".
Click "Add pool" to create a new agent pool and give it a name.
Here I have added my machine to the Default pool
Download and Configure the Agent:
Within the agent pool, click on the "New agent" button.
Choose the operating system of your machine and download the agent package.
Extract the downloaded package to a directory on your machine.
Open a command prompt or terminal and navigate to the directory where you extracted the agent.
Run the Configuration Script:
Execute the config.sh script (for Linux/macOS) or config.cmd script (for Windows) and follow the prompts:
./config.sh
Provide the server URL (Azure DevOps organization URL).
Enter a Personal Access Token (PAT) with sufficient permissions to register the agent.
Follow the prompts to configure the agent, including setting the agent pool name and agent name.
Run the Agent:
After configuration, start the agent using the provided command:
./run.sh
For Windows, use config.cmd and svc.cmd accordingly.
Verify the Agent:
Go back to Azure DevOps and verify that the new agent appears in the agent pool and is online.
