GitLab CI/CD: Setup, Pipeline Configuration, and FOSSA Integration
Technical guide on setting up GitLab CI/CD pipelines, covering .gitlab-ci.yml configuration, runners setup, deployment strategies, caching, security best practices, and integrating FOSSA for license compliance / vulnerability management.
Contents
Setting Up GitLab CI/CD from Scratch
GitLab CI/CD lets you automate builds, tests, and deployments by defining pipelines in a YAML file. Each project can have a pipeline that runs on every commit, orchestrated by GitLab Runner agents. This guide walks you through creating a pipeline configuration (.gitlab-ci.yml), setting up runners, and deploying to common environments (Docker, Kubernetes, cloud) with best practices for caching, security, and more.
1. Create the .gitlab-ci.yml Pipeline File
In your repository's root, add a file named .gitlab-ci.yml. This file defines your CI/CD pipeline stages and jobs. When you commit it, GitLab will detect it and trigger the pipeline automatically. Follow these steps to configure it:
Define Stages: List the pipeline stages in order (e.g., build, test, deploy). Jobs in the same stage run in parallel (if runners are available), and stages run sequentially.
Add Jobs: Create one or more jobs under each stage. Each job needs a unique name and at minimum a stage: and a script: to run. For example, you might have a build job to compile code and a test job to run tests.
(Optional) Specify Image & Variables: You can specify a Docker image for the job environment with the image: keyword, or set global variables: (like language versions or config flags) that all jobs can use.
Conditional Execution: Use rules to control when jobs run. For example, a deploy job should only run on the main branch or tags. GitLab's rules: (or legacy only/except) allow you to run or skip jobs based on branch, tags, or pipeline triggers.
Commit the File: Push the .gitlab-ci.yml to your repo. GitLab will create a new pipeline and execute the jobs as defined. You can monitor the pipeline's progress in the CI/CD > Pipelines section of your project.
For a complete understanding of all available GitLab CI/CD configuration options, refer to the GitLab CI/CD documentation.
Example: Basic .gitlab-ci.yml – The snippet below defines two stages (test and build) and two jobs. The test_job echoes a message and the build_job runs after tests, echoing a build message using a custom variable:
variables: APP_NAME: "demo" # example variable accessible in jobsstages: - test - buildtest_job: stage: test script: - echo "Testing $APP_NAME"build_job: stage: build script: - echo "Building $APP_NAME"
This pipeline will run the test_job, then the build_job. You can add more stages (like deploy) and jobs as needed. For instance, a job with environment: production would be treated as a deployment to the production environment. Remember that jobs in the same stage run in parallel and each job's script can have multiple shell commands.
2. Setting Up GitLab Runners
GitLab Runner is the agent that executes your CI jobs. GitLab.com provides shared runners by default for your projects, so you can often run pipelines without any setup. If you need a custom runner (for self-managed GitLab or specialized environment), follow these steps:
Install GitLab Runner: Download and install the GitLab Runner on your server or local machine (runners can also run in Docker).
Register the Runner: In your project's Settings > CI/CD > Runners, get the registration token. Run gitlab-runner register on your runner host and provide the GitLab URL, token, and a description. When prompted, choose an executor (e.g., shell for simple scripts or docker for containerized jobs).
Configure Tags (Optional): If you want the runner to pick up only specific jobs, assign it tags during registration (and add the same tags to jobs in .gitlab-ci.yml via the tags: key). Otherwise, it will pick up any untagged jobs.
Verify Runner Availability: After registration, the runner should appear in your project's Runners settings. An active runner shows a green status. Now when you push commits, the runner will execute your pipeline jobs.
Tip: If using Docker-in-Docker or building container images in jobs, ensure the runner is configured with privileged mode (needed for Docker commands). The GitLab documentation on using Docker to build Docker images provides detailed instructions for this setup.
3. Deploying to Common Environments
Once your build and test stages pass, you'll often want to deploy artifacts or applications. GitLab CI/CD is very flexible – you can deploy Docker containers, update Kubernetes clusters, or interact with cloud services all from your pipeline. Below are examples of how to deploy to Docker, Kubernetes, and cloud providers:
Deploying with Docker (Containers)
A common use-case is building a Docker image in CI and pushing it to a registry. To enable Docker commands in your jobs, your runner must allow it (e.g., using the Docker executor with the Docker-in-Docker service). In your .gitlab-ci.yml, you can use Docker's official images and services:
Docker Build and Push: Use a Docker image (like docker:latest) and add the Docker daemon service (docker:dind). For example, a job can specify image: docker:latest and services: [docker:dind]. Before building, log in to your container registry. You can use GitLab's built-in variables for its registry – for instance, echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin will log in to GitLab's Container Registry. Then run docker build and docker push commands in the script to build the image and push it to the registry.
Example: A build stage job might build an image and push to GitLab's registry. The image tag can include the commit SHA ($CI_COMMIT_SHA) or another unique identifier. Best practice: avoid using the latest tag for concurrent builds to prevent conflicts; instead, tag with version or commit ID.
After the image is pushed, you can deploy it. For instance, you might run a container on a server or update a Kubernetes deployment to use that new image. The CI job can trigger those actions (via SSH, Docker commands, or Kubernetes as shown below).
GitLab CI can interface with Kubernetes clusters to deploy applications. You have two main options: use GitLab's Kubernetes integration (the GitLab Agent) or use direct kubectl/Helm commands in the CI job. In both cases, you'll need to provide access credentials to the cluster (via the agent or kubeconfig/credentials in variables).
GitLab Kubernetes Agent: You can install an agent in your cluster and connect it to your GitLab project. This allows your CI jobs to authenticate to the cluster securely without embedding credentials. When using the agent, set the KUBECONTEXT for the job to the agent's context and run Kubernetes commands. For example, GitLab allows you to run kubectl apply or helm upgrade in CI against your cluster in a secure way. Using the agent, you might have a deploy job like:
deploy_to_k8s: stage: deploy image: bitnami/kubectl:latest # an image with kubectl variables: KUBECONFIG: "$CI_PROJECT_DIR/config" # if using a kubeconfig file from variables script: - kubectl apply -f k8s-manifest.yml environment: production
This assumes you have a k8s-manifest.yml (or Helm chart) in your repo and a KUBECONFIG file or agent context set up for access.
Direct kubectl with Credentials: Alternatively, without the agent, you can store a Kubernetes config or token as a CI/CD variable. For example, save a base64-encoded KUBECONFIG file as a protected variable and in your job script decode it to a file, then run kubectl. The above example shows one way: mounting a kubeconfig via variables. Ensure your cluster credentials are kept secret (use protected & masked variables).
Example: A deploy job might simply run kubectl apply -f deployment.yml to apply a Kubernetes manifest. This would update or create Kubernetes resources (like Deployments or Services) as defined in that file. By marking the job with environment: production or another environment name, GitLab will track the deployment in the Environments dashboard.
GitLab CI/CD can deploy to cloud services by using CLI tools or API calls. The general strategy is to use the cloud provider's CLI within a CI job and provide authentication via environment variables or CI/CD secret variables:
Setup Cloud Credentials: In your GitLab project settings, add CI/CD variables for your cloud credentials (for example, for AWS you'd add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). Mark these variables as protected (so they only run on protected branches like main) and masked (so they don't leak in logs).
Use CLI Docker Image: Use an official CLI image for your provider in the job. For instance, to deploy to AWS, you can use the Amazon AWS CLI Docker image and run AWS commands. In your job, set image: amazon/aws-cli (with an empty entrypoint) so you can directly use the aws command. Similarly, for Google Cloud you might use google/cloud-sdk image, or for Azure use the Azure CLI image.
Example (AWS S3 Deployment): Suppose you want to upload a static site to an S3 bucket. You could add a job:
Here, $S3_BUCKET is a variable storing your bucket name. The job uses AWS credentials from the environment to authorize the aws s3 sync command. This will sync the public directory with the S3 bucket (deleting removed files) – effectively deploying your site. Using the official AWS CLI image means you don't need to install anything extra in the job environment.
No matter the environment, always test your deployment steps locally or in a staging environment first. You can use GitLab CI environments (like environment: staging vs production) to separate deployments. Also consider using GitLab's Environments and Deploy Boards features for Kubernetes and review apps if applicable.
4. Best Practices for CI/CD Pipelines
To make your pipelines efficient and secure, consider the following best practices:
Caching Dependencies: Leverage caching to speed up builds. Define a cache: in jobs for package directories (e.g., node_modules/, vendor/) so that subsequent jobs or pipelines reuse downloaded dependencies. Use cache keys smartly – for example, include the branch name or lockfile (Gemfile.lock, package-lock.json) in the key to bust cache when dependencies change. For maximum cache availability, use a consistent runner or enable distributed caching (e.g., an S3-backed cache for autoscaled runners) so all jobs can access the cached content. The GitLab CI Caching documentation provides comprehensive guidance on effective caching strategies.
Security and Secrets: Do not store sensitive credentials in the repo or in plain text in the .gitlab-ci.yml. Use CI/CD variables for secrets like API keys, and mark them masked and protected. Protected variables will only be available on protected branches or tags (e.g., your main or release branches) to prevent exposure from untrusted forks. For extremely sensitive secrets, consider using external secret managers (like HashiCorp Vault or cloud key management services) which GitLab can integrate with. Also, pin your dependencies and base images to specific versions or SHAs – for example, use a specific Docker image digest instead of latest to avoid unpredictable changes. This improves both security and build reproducibility.
Pipeline Optimization: Keep pipelines fast and efficient. Parallelize independent jobs and tests to run simultaneously and reduce total time (GitLab supports a parallel: keyword or simply multiple jobs in the same stage). For example, split a large test suite into parallel jobs (sharded by $CI_NODE_INDEX) to finish quicker. Use smaller base images for jobs to cut down startup time – e.g., an Alpine-based image can speed up a job compared to a full Ubuntu image. Only run jobs when needed: use rules: or only: to skip jobs on certain branches or conditions (for instance, only run deployment jobs on the main branch, and skip on merge request branches). Arrange your stages so that quick feedback (linting/tests) comes early, and heavier deploy steps come last, ensuring failures are caught early. For more optimization tips, check out the GitLab CI Optimization guide.
Modular Configuration: As your pipeline grows, avoid a monolithic config. GitLab allows you to include other YAML files in your .gitlab-ci.yml. You can split jobs into multiple files (for example, have a separate tests.yml, deploy.yml, etc.) and include them, or reuse shared pipeline templates across projects. For instance, use include: 'common-ci.yml' to bring in common job definitions. This keeps the main .gitlab-ci.yml concise and lets you maintain reusable configs. You can even include predefined templates provided by GitLab (e.g., template: Auto-DevOps.gitlab-ci.yml) or configs from other projects. Modularizing your pipeline makes it easier to manage and scale. Learn more in the GitLab CI include documentation.
By following these practices – caching files, securing secrets, optimizing job runtimes, and keeping configurations DRY – you'll have a faster, safer, and more maintainable GitLab CI/CD pipeline. With your runners configured and .gitlab-ci.yml in place, you're ready to automate your software lifecycle from commit to deployment 🚀. Enjoy the continuous integration/deployment workflow with GitLab!
5. Integrating FOSSA with GitLab CI/CD
Integrating FOSSA into your GitLab CI/CD pipeline enables automated license compliance and vulnerability scanning for your dependencies. This ensures that problematic licenses or security vulnerabilities are detected early in the development process.
Setting Up FOSSA Integration
To integrate FOSSA with GitLab CI/CD:
Obtain a FOSSA API Key: Generate an API key from your FOSSA account settings.
Add the API Key as a CI/CD Variable: In your GitLab project, go to Settings > CI/CD > Variables and add a new variable named FOSSA_API_KEY with your API key. Mark this variable as masked and protected.
Add FOSSA to Your Pipeline: Include a FOSSA analysis job in your .gitlab-ci.yml file, typically after dependencies are installed:
