Skip to main content
FOSSA Logo

CircleCI Setup and Usage Guide

Technical documentation for setting up, configuring, and using CircleCI pipelines with GitHub, GitLab, and FOSSA integration

Contents

About

CircleCI is a continuous integration and delivery (CI/CD) platform that automates the build, test, and deployment of software. It integrates with popular version control systems – GitHub, GitLab (both SaaS and self-managed), and Bitbucket – to trigger pipelines on code commits. You can run CircleCI pipelines in cloud-hosted environments or on your own infrastructure (CircleCI Server). There's also a local CLI for running jobs on your machine and supporting advanced workflows.

Key benefits: CircleCI is known for fast, efficient pipelines that can be optimized with caching and parallelism. It supports complex workflows and orbs (reusable configuration packages) to integrate with third-party tools and services. Developers can SSH into jobs for debugging, set up parallel test runs, and leverage Docker layer caching to speed up container builds. CircleCI provides first-class support for a variety of executor types (Linux Docker containers, Linux VMs, macOS, Windows) and resource classes for customizing compute power. Typical use cases include continuous integration of web/mobile applications, running automated test suites on each pull request, building and pushing Docker images, and orchestrating deployments upon successful builds.

You can learn more about available VCS integrations in the CircleCI documentation on version control system integration.

CircleCI Setup Guide

Account Setup (CircleCI with GitHub and GitLab)

Getting started with CircleCI involves creating an account and connecting your code repository. Sign up on the CircleCI platform (using your email or via OAuth) and authorize CircleCI to access your repositories on GitHub or GitLab. If you choose the GitHub integration, you will install the CircleCI GitHub App and select which repositories to grant access to; for GitLab, you will connect CircleCI to your GitLab instance and authorize access.

Once connected, add a project in CircleCI by selecting the repository. CircleCI will look for a .circleci/config.yml file in the repo's default branch. If a config file is missing, CircleCI can provide a starter config template (you can download or have CircleCI auto-push it). After the project is created and a config is in place, any new commit will trigger a pipeline.

The CircleCI blog provides a detailed guide on setting up continuous integration with GitHub that walks through each step of this process.

(CircleCI supports both its cloud SaaS service and a self-hosted Server edition. Ensure you've chosen the appropriate option for your needs; see next section.)

CircleCI Cloud vs CircleCI Server

CircleCI is offered in two modes of deployment: Cloud (hosted by CircleCI) and Server (self-hosted on your infrastructure). In CircleCI Cloud, all infrastructure setup and maintenance is managed by CircleCI – you get instant access to new features and automatic upgrades with no server management overhead. This is ideal for most teams, as it allows quick setup and scaling without worrying about installation. In CircleCI Server, you install CircleCI on your own servers or cloud environment (e.g. AWS or GCP) and manage it yourself. Server mode keeps the data and build runners behind your firewall, offering greater control and compliance for enterprises, at the cost of manual maintenance and delayed feature updates.

Pros & Cons:

  • Cloud: Zero maintenance, immediate updates, and easy onboarding. Limited by internet access (your code is processed in CircleCI's cloud) and subject to usage quotas/credit consumption.
  • Server: Complete control over environment and data (meets strict security or air-gapped requirements). However, you must provision and update the CircleCI installation, and scaling requires managing your own compute resources.

Choose CircleCI Cloud for a hassle-free SaaS experience, or CircleCI Server if your organization requires on-premises builds or custom networking setups.

For more information on deployment options, check the CircleCI FAQ about cloud vs server deployments.

Installing the CircleCI CLI

For advanced workflows and local development, CircleCI provides a CLI tool. The CircleCI CLI allows you to validate config files, run jobs locally (with Docker), and interact with the CircleCI API. Install it via your preferred package manager or script:

  • macOS: Use Homebrew – for example, brew install circleci
  • Linux: Install via Snap: sudo snap install circleci (installs the CLI in an isolated environment along with Docker). Alternatively, use the official install script: curl -fLSs https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/main/install.sh | bash
  • Windows: Use Chocolatey – e.g. choco install circleci-cli -y

After installation, run circleci setup. This will prompt you for a CircleCI API token and the host (choose the default for CircleCI cloud, or your server URL for CircleCI Server). Generate a personal API token from the CircleCI UI (User Settings → Personal API Tokens) and paste it when prompted. Once configured, you can use commands like circleci local execute to run a job from your config locally, or circleci config validate to lint your config file before pushing changes.

The CircleCI local CLI documentation provides complete installation instructions for all platforms.

Writing Your First CircleCI Pipeline Configuration

Every CircleCI project is driven by a YAML configuration file, typically located at .circleci/config.yml in your repo. This file declares your pipelines, which are composed of workflows and jobs. At minimum, a CircleCI config defines one or more jobs (each job is a collection of steps to execute in a given environment), and a workflow to orchestrate those jobs. Below are simple examples of a CircleCI config for three different languages (Node.js, Python, Go) to illustrate the structure:

.circleci/config.yml (Node.js example)
version: 2.1
jobs:
  build:
    docker:
      - image: cimg/node:18.16  # CircleCI Node.js image
    steps:
      - checkout
      - restore_cache:
          key: node-deps-{{ checksum "package-lock.json" }}
      - run: npm install
      - save_cache:
          key: node-deps-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
      - run: npm test
workflows:
  node_pipeline:
    jobs:
      - build
.circleci/config.yml (Python example)
version: 2.1
jobs:
  test:
    docker:
      - image: cimg/python:3.10   # Use CircleCI Python image with desired version
    steps:
      - checkout
      - restore_cache:
          key: py-deps-{{ checksum "requirements.txt" }}
      - run:
          name: Install dependencies
          command: |
            python -m venv venv
            . venv/bin/activate
            pip install -r requirements.txt
      - save_cache:
          key: py-deps-{{ checksum "requirements.txt" }}
          paths:
            - venv
      - run: 
          name: Run tests 
          command: |
            . venv/bin/activate
            pytest
      - store_artifacts:
          path: test-results/
          destination: python_tests
workflows:
  py_pipeline:
    jobs:
      - test
.circleci/config.yml (Go example)
version: 2.1
jobs:
  build-and-test:
    docker:
      - image: cimg/go:1.20   # CircleCI Go image
    steps:
      - checkout
      - restore_cache:
          key: go-mod-{{ checksum "go.sum" }}
      - run: go mod download
      - save_cache:
          key: go-mod-{{ checksum "go.sum" }}
          paths:
            - "~/.cache/go-build"
            - "~/go/pkg/mod"
      - run: go test -v ./...
workflows:
  go_pipeline:
    jobs:
      - build-and-test

In these examples, each job uses a CircleCI convenience image (pre-built Docker images for common languages). We use checkout to pull down the source code, then restore dependency caches if available (using a key that fingerprints dependency files like package-lock.json or go.sum). Next, we install dependencies and save them to cache for future runs, run the build/tests, and optionally store artifacts (like test result files). The workflows section triggers the job. This configuration-as-code approach means your CI pipeline is versioned alongside your application code.

Refer to the CircleCI configuration reference for the full syntax and options.

Config tip: CircleCI configuration is very flexible. You can define multiple jobs (for build, test, deploy, etc.) and orchestrate them in the workflow (even run some in parallel). You can also use orbs to simplify config for common tasks (for example, the Slack orb for notifications, or language-specific orbs to install dependencies). The examples above hard-code steps, but CircleCI provides official orbs (like circleci/node, circleci/python) which can reduce boilerplate.

Running Your First Pipeline and Debugging Tips

Once your config is defined and pushed to the repository, CircleCI will automatically trigger a pipeline for any new commit on the configured branch. You can monitor progress on the CircleCI web app. Pipelines consist of workflows and jobs; the CircleCI dashboard provides real-time logs and status for each job.

CircleCI Pipelines dashboard, listing recent pipelines and their status for several projects. The web interface allows you to drill down into each workflow and job to see console output, timing, and status. You can quickly rerun failed workflows or jobs with a click (with the option to enable SSH for debugging).

For failed jobs, CircleCI enables an "SSH into job" feature that reruns the job and drops you into a live shell in the failed container, so you can inspect the environment and troubleshoot issues. This is extremely useful for debugging complex failures in situ. Additionally, you can download any artifacts or logs that the job saved, which helps in analyzing test failures or build outputs.

If a pipeline doesn't trigger as expected, ensure that the project is following the correct branch and that the config file is valid. The circleci config validate command (or the "Config Processing" section in the CircleCI UI) can help identify YAML syntax errors. Common first-run issues include the project not being set up on CircleCI, missing config file, or incorrect indentation in YAML.

For more information on troubleshooting pipeline triggers, see the CircleCI documentation on pipelines and triggers.

Ongoing Usage

After the initial setup, you'll want to leverage CircleCI's features to optimize build performance and maintain robust pipelines. This section covers caching, parallelism, artifacts, and other best practices to keep your CircleCI CI/CD pipelines fast and efficient.

Caching Strategies

Effective caching can dramatically speed up your pipelines. CircleCI allows you to cache dependencies or other build outputs between runs using the save_cache and restore_cache steps. By reusing data from previous jobs, you avoid re-downloading packages or rebuilding assets on every run, saving time and compute resources. Some tips for caching:

  • Dependencies caching: Cache language dependencies (e.g., node_modules, Python virtualenv, Go modules) keyed by a checksum of your lockfile (as shown in the config examples above). This ensures the cache is invalidated when dependencies change, but reused if they remain the same.
  • Avoid overly specific keys: Don't include variables like exact commit SHA in your cache key, or you'll miss out on cache hits. Using a rolling cache key that's too unique (e.g., the commit hash) means the cache is almost never re-used. Instead, use broader keys (like dependency file checksum or branch name) to maximize hits.
  • Partial caches: It can be useful to split caches by category (for instance, separate caches for frontend and backend dependencies in a monorepo) to avoid invalidating everything on a small change.
  • Cache persistence: By default, caches are kept for a long time, but you can control retention in CircleCI's settings if needed (to save storage). Periodically updating a version prefix in your keys (e.g., v1-... to v2-...) can force refresh when necessary (for example, after a major dependency upgrade).

CircleCI restores caches at the start of a job and saves them at the end, so plan your steps accordingly (install steps should occur between restore and save). Proper caching is often the easiest way to cut down build times.

For detailed guidance on optimizing your caching strategy, check out the CircleCI caching strategies documentation.

CircleCI Docker Layer Caching

If your workflow builds Docker images, Docker Layer Caching (DLC) can significantly reduce build times. CircleCI's Docker executor normally does not persist Docker cache between jobs, but CircleCI offers DLC as an option on certain plans. With DLC enabled, Docker image layers are saved and reused on subsequent runs, so unchanged layers are not rebuilt each time. This is especially beneficial for large images or projects with multi-stage Dockerfiles.

To use Docker layer caching, you may need to enable the feature in your project settings or use a special executor (for example, the machine executor with the docker_layer_caching option, available on paid plans). When configuring DLC, ensure that your jobs use the setup_remote_docker step with the docker_layer_caching flag if required. According to CircleCI, using DLC can speed up container builds by reusing up to 30–80% of layers, depending on the changes between builds. Keep in mind DLC consumes additional credits and may have cost implications on the cloud plan, but for heavy Docker workflows it often pays for itself in time saved.

Parallelism and Matrix Builds

CircleCI can run multiple tasks in parallel, which is key for speeding up CI/CD pipelines. You can use parallelism within a job to fan out a single job into multiple executors (for example, split test files across 4 parallel instances to cut overall test time). CircleCI provides an environment variable (CIRCLE_NODE_INDEX) in parallel executions to differentiate between them, and features like automatic test splitting to divide work optimally. For instance, if you have 1000 tests that take 10 minutes, running 5 parallel instances could potentially bring the test stage down to ~2 minutes by distributing tests.

In addition, CircleCI supports matrix builds (matrix jobs) which allow you to declaratively run a job with multiple variations (e.g., test against multiple language versions, OSes, or dependency versions). Instead of writing separate jobs, you can define a job with parameters and use the matrix strategy to generate a combination of jobs. For example, you might test a library against Python 3.8, 3.9, and 3.10 in one matrix job, or build a container on linux/amd64 and linux/arm64 variants. Matrix jobs simplify cross-environment testing and ensure broad compatibility. They are configured by adding a special matrix section under a job in the workflow. CircleCI then expands the matrix, running one job per combination of parameters. This helps achieve broad test coverage without duplicating config.

When using parallelism or matrices, monitor your runtime and adjust as needed – oversharding (too many parallels with too little work each) can lead to diminishing returns or increased overhead. Use CircleCI's Insights to see how long jobs take and find the optimal parallelism factor.

Artifacts and Workspaces

Artifacts and workspaces are two methods to persist data in CircleCI beyond a single job. Use artifacts to save files from your jobs so that you can access them after the pipeline finishes (for example, test reports, coverage results, build binaries, screenshots, etc.). To store an artifact, add a store_artifacts step specifying the file or directory path. After the job completes, these artifacts are uploaded and made available in the CircleCI web UI and via API. This is useful for surfacing test results (which can be integrated with GitHub via checks), or for keeping build outputs (like compiled binaries or deployment packages) for later download. Artifacts persist after pipeline completion, but note they have a default retention period (usually 30 days on CircleCI cloud).

Workspaces are used to pass files between jobs in the same workflow. If you split your pipeline into multiple jobs (say one job builds an application, a later job deploys it), you can use workspaces to transfer the build output from the build job to the deploy job. A job can persist to workspace specific files or directories at the end of its run, and downstream jobs can attach the workspace at the start to retrieve those files. Workspaces are scoped to a workflow run and discarded afterward. They enable breaking pipelines into logical stages without re-doing work.

For example, a build job could compile code and persist the dist/ directory to a workspace, then a test job could attach that workspace to run tests on the compiled artifacts, followed by a deploy job that takes the same artifacts to push to production. Using workspaces avoids rerunning the build in each job and ensures consistency across stages.

Learn more about this approach in the CircleCI documentation on using workspaces to share data between jobs.

In summary, artifacts are for exporting data out of the pipeline (to users or external systems), and workspaces are for sharing data between jobs within the pipeline. Both can be combined with caching: cache for dependencies between separate pipeline runs, workspaces for data within a single pipeline run, and artifacts for long-term storage or debugging.

Integrations and Notifications

CircleCI's extensibility allows it to integrate into your broader development workflow:

  • VCS Status Checks: When CircleCI runs on a GitHub or GitLab repository, it reports build statuses back to the platform. For GitHub, you'll see checks or status messages (pass/fail) on commits and pull requests, so you can require passing CircleCI checks before merge. For GitLab, pipelines show up natively in the merge request interface. This tight integration ensures that CI results are visible where code reviews happen.
  • Slack Notifications: Getting notified of pipeline results is easy with CircleCI's Slack integration. The recommended approach is to use the CircleCI Slack orb, which allows you to send messages to Slack channels based on job or workflow events (success, failure, fixed, etc.). By adding a few lines in your config (invoking circleci/slack@x.y orb and using its notify command), you can receive alerts when a build fails or when a deploy succeeds. This keeps the team informed in real-time. (Remember to add the Slack webhook or OAuth token as a secure environment variable as required by the orb configuration.)
  • Other integrations: CircleCI offers a rich orb registry and API. You can integrate with testing services (e.g., coverage tools like Codecov), deployment targets (AWS, Google Cloud, Kubernetes), monitoring (Datadog), and more by installing orbs or calling external APIs from your pipeline. For example, there are orbs for sending GitHub commit statuses, creating JIRA tickets on failures, or running security scans. CircleCI's webhook and API also let you trigger pipelines or get notifications in custom ways. Many third-party CI/CD integrations (like FOSSA, as we'll discuss next) are implemented simply by adding a CLI invocation in a job or using an orb if available.

CircleCI CI/CD Best Practices

To get the most out of CircleCI, consider these best practices that experienced teams employ:

  • Reusability: Don't repeat yourself in config. Use orbs and reusable commands/executors. CircleCI orbs (both official and community) encapsulate common patterns, so you can import an orb rather than writing complex bash scripts for every pipeline. You can even create private orbs within your org to share pipeline components across projects.
  • Optimize build feedback time: Aim to fail fast and surface feedback quickly. Run faster linters or unit tests early in the pipeline, and parallelize where possible to reduce overall time. Utilize test splitting to parallelize large test suites automatically.
  • Caching and performance: As discussed, caching is critical. Also consider using smaller base images or custom Docker images preloaded with dependencies to cut down setup time. Monitor your job durations via CircleCI Insights and identify bottlenecks.
  • Security and secret management: Store sensitive values (API keys, credentials) as environment variables or contexts in CircleCI, rather than hardcoding them. CircleCI masks these in logs and secures them. Use contexts to limit secret access to the jobs that need them, and leverage features like restricted contexts for production deploys (only certain users can trigger) to add security gates.
  • Branch-specific workflows: Tailor your pipelines for different branches – e.g., run a full test and deploy workflow on the main branch, but only run lint and unit tests on feature branches. You can use workflow filters or pipeline parameters to do this. This helps to save time and credits by not running unnecessary jobs on every push.
  • Fail loudly and early: If using multiple jobs, have a strategy for failures. By default, if any job fails, the workflow fails. You can make use of the when clause to conditionally run jobs (for example, only run deployment job when tests pass). Ensure your pipeline surfaces errors clearly – e.g., use the error step to explicitly mark known failure conditions.
  • Leverage Insights and Metrics: CircleCI provides an Insights dashboard per project to see trends in pipeline success, durations, and credit usage. Use these metrics to continuously improve. For instance, if a particular job often flakes or slows down, you might invest in splitting it or increasing its resource class. If build times creep up, revisit caching strategy or dependency bloat.
  • Keep configurations in version control: Treat the CI config as code – use pull requests to modify .circleci/config.yml, so changes to pipelines are reviewed. This reduces the chance of introducing breaking changes and ensures auditability of any pipeline changes.

For a comprehensive overview of CI/CD best practices for CircleCI, check out the CircleCI blog on top 5 CI/CD best practices.

By adhering to these best practices, engineering teams (including large-scale projects) can maintain fast, reliable pipelines that scale with their codebase. CircleCI's flexibility and power make it possible to achieve very high throughputs – some organizations run thousands of workflows a day on CircleCI by optimizing according to these principles.

Integrate FOSSA with CircleCI

Modern development not only requires testing your own code, but also scanning your dependencies for licensing and security risks. FOSSA is a tool that automates open source license compliance and vulnerability scanning. You can seamlessly integrate FOSSA into your CircleCI pipeline to catch license or security issues during the CI process, ensuring continuous compliance.

Setup: To integrate FOSSA with CircleCI, first obtain your FOSSA API key. In your FOSSA account settings, create an API token (consider using a "Push Only Token" for public projects). In CircleCI, add this token as an environment variable named FOSSA_API_KEY in your project settings (so it will be available to jobs at runtime).

For detailed integration instructions, visit the FOSSA documentation for CircleCI integration.

flowchart LR
    developer((Dev)) --> push[Code Commit]
    push --> ci_pipeline((CircleCI Pipeline))
    ci_pipeline --> build_step[Build & Test]
    build_step --> fossa_scan{FOSSA Scan}
    fossa_scan -- "no issues" --> deploy_step[Deploy to Environment]
    fossa_scan -- "issues found" --> fail_stop[Fail Pipeline]
    style fossa_scan fill:#f9f,stroke:#333,stroke-width:2px

Mermaid diagram: a developer's commit triggers a CircleCI pipeline. After build & test steps, a FOSSA scan runs. If FOSSA reports license or security issues, the pipeline fails, otherwise deployment proceeds.

Integrating FOSSA in CircleCI provides automated license compliance checks and vulnerability scanning as part of your CI/CD workflow. This proactive approach ("shift left" for compliance/security) ensures that problematic dependencies are flagged early. FOSSA's analysis covers all transitive dependencies and can enforce policies (for example, to approve use of certain licenses) (Open Source License Compliance Management | FOSSA) (Open Source License Compliance Management | FOSSA). By the time your code reaches later stages or gets deployed, you have high confidence that you are not introducing unknown legal risks or critical vulnerabilities. The continuous compliance achieved by CircleCI + FOSSA integration means no more last-minute surprises about an open-source component's license or a library that needs a security patch – those issues are caught at build time (Open Source License Compliance Management | FOSSA).

In summary, CircleCI's powerful CI/CD automation combined with FOSSA's open-source risk management gives your team an efficient way to ensure every build is vetted for OSS license and security issues. This integration is just one example of extending CircleCI with specialized tools – thanks to CircleCI's flexibility, adding such steps is straightforward, and it enables a robust DevSecOps practice out-of-the-box.