Action

Week 0x8 II

Prof. Calvin

Crypto

Announcements

  • Welcome to variously CS 276/CS 540
    • Back to containers.
    • Week off from C
    • Two lectures with exercises
  • Action Items:
    • BigRSA due 31 Mar

Today

  • DevOps
  • Continious Integration
    • make
    • GitHub Actions

Motivation

  • Last class we had this extremely cursed way of doing things.
    • We used a bunch of complicated and hard to remember podman commands.
    • After those commands, we supplied further bash or sh shell commands.
    • It was a hassle and a headache I don’t like it.

Motivation

  • Last class we uncovered a cool new way to do things.
    • We can do in-container development externally to a container.
    • We can:
      • Build
      • Test
    • This is the modern dev tech stack.

CI

  • For Continuous Integration
    • Modern approach to combined unit, build, and integration testing.
    • Usually done on build servers
    • Pioneered by Google et al. for web services
  • Component of DevOps and the more common term CI/CD

CI/CD Introduction

  • Continuous Integration (CI) and Continuous Delivery (CD).
    • Automates software development processes.
      • Push code to repo
      • Repo builds, tests, and deploys software
        • Either as packages for download or a live service.
    • Improves software quality and delivery speed [Citation Needed]

Editorializing

  • I was an extreme software engineering skeptic
  • I believe virtually all claims of software engineering advancements have been proven false by:
    • Collapse of Java and emergence of Rust when devs WFH in 2020
    • Tech buildouts in China and India
    • And before that, FOSS

The Alternative

  • To me, CI/CD is the demonstrably correct approach to development.
  • Scientifically proven by the Linux kernel
    • FOSS
    • Higher quality than Windows/Mac and their \(n\)-trillion dollar sponsors
    • Completely imperative (no objects)
    • No managers/No offices/No sprints

See also

  • GCC
  • FreeBSD
  • NGINX
  • HTTPD
  • PostgreSQL

Both Sides

  • I have actually never heard a counterargument to this hypothesis.
  • I think the actual counterargument is “a lot of people make money doing waterfall and agile”
    • I am arguing from a correctness rather than profitability perspective.

Case Study

As part of the “Get the Facts” campaign, Microsoft highlighted the .NET Framework trading platform that it had developed in partnership with Accenture for the London Stock Exchange, claiming that it provided “five nines” reliability. After suffering extended downtime and unreliability the London Stock Exchange announced in 2009 that it was planning to drop its Microsoft solution and switch to a Linux-based one in 2010.

Mason, Rowena (September 10, 2008). “Seven-hour LSE blackout caused by double glitch”. The Telegraph. London. Archived from the original on December 25, 2017. Retrieved April 3, 2018.

“London Stock Exchange trading hit by technical glitch”. BBC News Online. BBC. November 26, 2009. Archived from the original on July 26, 2013. Retrieved September 30, 2010.

Williams, David M. (October 8, 2009). “London Stock Exchange gets the facts and dumps Windows for Linux”. ITWire. Archived from the original on July 16, 2011. Retrieved September 30, 2010.

“London Stock Exchange Rejects .NET For Open Source”. Slashdot. October 6, 2009. Archived from the original on August 11, 2011. Retrieved September 30, 2010.

Azure Linux

  • Since 2020 Microsoft just maintains a Linux distribution.

Azure Linux is an internal Linux distribution for Microsoft’s cloud infrastructure and edge products and services.

  • Under CI/CD GitOps on GitHub, the same site as the Linux kernel
  • Which Microsoft purchased in Jun 2018

Build, Test, Deploy

CI

  • Build and Test.
    • Ensures code changes integrate smoothly.
    • Detects errors as soon as they are written.
    • Can facilitate collaboration within a broader framework.
      • GitHub uses issues and pull requests
      • Historical, mailing lists (still in use of course).

Mailing List Ex.

So this email is not about some “Rust policy”. This email is about a much bigger issue: as a maintainer you are in charge of your code, sure - but you are not in charge of who uses the end result and how.

CD

  • Not cd and not “Crof. Dalvin”
  • Not really part of this class.
    • Our code is banned for use.
  • Automated deployment to staging/production.
    • Sort of like allowing me to see your code while you work on it.
  • In a way, CD from cd to C.D.

What is DevOps?

  • Combines software development (Dev) and IT operations (Ops).
  • Shorten the development lifecycle.
  • Collaboration, automation, integration, and rapid feedback cycles.

What does it mean?

  • Integrate development (gcc) and operations (ubuntu).
  • Automating as many tasks as possible.
  • Using tools to manage the code (not people).

DevOps & CI?

  • Continuous Integration (CI) is the practice of automating the integration of code changes into a shared repository.
  • By using CI, we will be users of the Dev part of DevOps
  • Clients of the Ops part (which is docker/podman)

GitHub and CI/CD

  • GitHub Actions for automation.
    • Workflow definitions in YAML files.
      • I’d call these “scripts”
    • Integration with GitHub repositories.
    • Support for various programming languages.
      • Including C!

Workflows

  • Hidden .github/workflows folder in repos:
    • Triggers: Events that start a workflow.
    • Jobs: Sets of steps executed on a runner.
      • Steps: Individual tasks within a job or lines in a script.
    • Runners: Virtual machines or containers.

Example CI Workflow

  • Checkout code from repository (into container)
    • Install dependencies.
      • Or use a base image.
    • Run unit tests.
    • Build application artifacts (logs)

Example CD Workflow

  • Deploy application to GitHub Pages
    • Run integration tests.
    • Deploy to live.
  • Using this for the website for now.
    • Probably needs work, I don’t love it ATM.

Smaller Ex.

Excerpt

  • Here’s a line under the “Build with Jekyll” heading:
  • Folks… it’s containers all the way down.
    • That’s a Linux script running docker 😂
    • Would’ve been podman before 2018 I bet.

Back in Action

  • GitHub Pages is a little bit automated
    • Read: I taught it to first years who had never programmed.
    • I plan to teach it to high schoolers this summer.
  • As with C/Python, in this class we index into clarity at the expense of brevity.
  • So we write our own CI action.

CI

  • Defined in a .yml file
    • GitHub provides templates
    • I found the templates self-documenting and documentation unusuable
  • You can press buttons on GitHub (click ops) or use command line.
    • You’ll never guess which one I think you should do.

Ref .yml

.yml

ci/c-cpp.yml
name: C/C++ CI

on:
  push:
    branches: [ $default-branch ]
  pull_request:
    branches: [ $default-branch ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4
    - name: configure
      run: ./configure
    - name: make
      run: make
    - name: make check
      run: make check
    - name: make distcheck
      run: make distcheck

Simplify

ci/c89.yml
name: C89 CI

on: push

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4
    - run: make

Simplify

Line Description
name: C89 CI I didn’t check if names were required.
on: push Actions will run after a push to repo
  • After a colon, either a single option or a new line, indentation, and series of options.

Jobs

Line Description
jobs: Job is like a .sh file
build: Usually one of build, test, deploy
runs-on: ubuntu-latest That is GitHub managed image
  • I tried to change ubuntu-latest to ubuntu and it broke.
    • There will always be latest, or fix one of the recent versions.

Runners

  • GitHub runners are distinct from the docker.io and other images.
  • Read more
  • Right now, here are the Linux options:
    • ubuntu-latest, ubuntu-24.04, ubuntu-22.04, ubuntu-20.04

Vs Docker

  • These are the mostly the same images available via dockerhub.
  • GitHub workflows just doesn’t bring over “rolling” and “devel” images.

Steps

Line Description
steps: Steps are like individual commands
- uses: actions/checkout@v4 This is some built-in you can’t change.
- run: make Runs make inside ubuntu-latest after checking out the repository within that container.

On .yml

  • You may have learned .yml from e.g. Data Engineering
  • It is used in the header to these slides:
    • je suis développeur web
action.qmd
---
title: Action
theme: dark
author: Prof. Calvin
subtitle: "Week 0x8 II"
institute: Crypto
format: 
    revealjs: 
        code-fold: true
        theme: dark
        mainfont: monospace
        slide-number: true
        show-slide-number: all
        width: 1050
        height: 700
        footer: "[Home](../index.html)"

execute:
    echo: true
    cache: true
    freeze: true  # never re-render during project render
    code-fold: false
---

Up to you

  • You can fiddle around with GitHub workflows and actions and etc and etc, it’s not too bad, but…
  • I’d use the above as boilerplate and only change one thing:
  • make

Today

  • DevOps
  • Continious Integration
    • make
    • ✓ GitHub Actions

make

  • make is the historically supported way to get things done related to C (and C++) development.
    • CMake is the new technique which is a bit heavier weight and used in larger projects.
      • By new I mean 2000.
      • make is 1976

The Pitch

  • make
    • Stores big commands
    • Can check how recently files have changed
    • Is programable via “rules” in a Makefile.

Example

  • Here’s a sample Makefile, the thing that programs make
    • Guess what Containerfile naming scheme is derived from!
    Makefile
    build:
  gcc shainc.c
  • Using SHA as the running example…

Example

  • Suppose we have an incomplete “shainc.c”
shainc.c
int main() {
        printf("hi\n");
}
  • We get some warnings:

Example

  • Given a Makfile that contains the same “rule”
Makefile
build:
    gcc shainc.c
  • We get the same warnings:

Takeaways

  • Make runs shell commands.
  • They run the same as if we type them ourselves.
  • This allows us to store commands.
  • Rules can be goofy - but by default the first one runs - we have one, named build, so it’s the one that happens.
Makefile
build:
    gcc shainc.c

Rules

  • You can also specify rules if you have more than one:
Makefile
build:
    gcc shainc.c

nowarn:
    gcc shainc.c -w

Use rules

  • We can “build”
  • Or we can specify “nowarn”
  • The first is the default

In practice

  • I tend to use a Makefile like this:
Makefile
build:
    gcc shainc.c --std=c89 -Wall -Wextra -Werror -Wpedantic -O2 -o shainc

clean:
    rm shainc
  • make will compile and make clean will remove the old executable if I don’t want it around for some reason.

Check

  • “make” commands tend to 5 char, next most common I see is “check”
  • We have a shell friendly shainc autograder!
tester.sh
gcc shainc.c -Wall -Wextra -Werror -Wpedantic -O2 -o shainc
echo "15 characters." > 15char.txt
echo "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum." > lipsum.txt
curl https://github.com/cd-public/books/raw/main/pg1342.txt -o austen.txt 2>/dev/null
echo " === Finding errors vs. reference implementation. === "
diff <(sha256sum 15char.txt) <(./shainc 15char.txt)
diff <(sha256sum lipsum.txt) <(./shainc lipsum.txt)
diff <(sha256sum austen.txt) <(./shainc austen.txt)
echo " === Errors printed. No errors denotes \"Perfect!\" === "

A note

  • This:
diff <(sha256sum 15char.txt) <(./shainc 15char.txt)
  • Is bash and only works via /bin/bash
  • Makefile uses sh which doesn’t allow the parenthesis like that.
  • We will use temporary files.

Step by step

  • This is just make build
gcc shainc.c -Wall -Wextra -Werror -Wpedantic -O2 -o shainc
  • These lines just echo text into a file:
echo "15 characters." > 15char.txt
echo "Lorem ipsum dolor sit amet, " # etc etc

Curl

  • This just pulls a book from GitHub using curl and usings the > character to direct all of the diagnostic information to nowhere.
curl https://github.com/cd-public/books/raw/main/pg1342.txt -o austen.txt 2>/dev/null
  • Try it with different flags if you are confused.

Curl

  • curl downloads a file from a url
  • -o specifies what to save that file as on the local computers
  • 2> suggests were to send error handling information
  • /dev/null is an imaginary file we can write to that won’t remember what we wrote there.
    • Like return none

Diff

  • diff compares two files to see if they are the same or different.
    • If they differ, it prints the difference and returns a non-zero value.
    • If they are the same, it prints nothing and returns zero.
    diff <(sha256sum 15char.txt) <(./shainc 15char.txt)
diff <(sha256sum lipsum.txt) <(./shainc lipsum.txt)
diff <(sha256sum austen.txt) <(./shainc austen.txt)

Wait - which files?

  • This is another imaginary file:
diff <(sha256sum 15char.txt) <(./shainc 15char.txt)
  • This is not allowed by make.
  • Instead, make a few files:
sha256sum 15char.txt > 15char.sum
./shainc 15char.txt > 15char.inc
diff 15char.inc 15char.sum

Try diff

  • Use echo to create 15char.txt and lipsum.txt.
  • Use diff to examine them:
$ diff 15char.txt 15char.txt
$ echo $?
0
$ diff 15char.txt lipsum.txt
1c1
< 15 characters.
---
> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
$ echo $?
1

Use /dev/null

  • Wait we don’t care what the difference is, just if there is one.
  • And we know how to ignore the output of a command.
$ diff 15char.txt 15char.txt >/dev/null ; echo $?
0
$ diff 15char.txt lipsum.txt >/dev/null ; echo $?
1
  • When files differ, the return value of diff suggests an error occured

Non-zero return

  • We get the same from gcc when a build fails.
$ gcc shainc.c -Wall -Wextra -Werror -Wpedantic -O2 -o shainc
shainc.c: In function ‘main’:
shainc.c:2:9: error: implicit declaration of function ‘printf’ [-Werror=implicit-function-declaration]
    2 |         printf("hi\n");
      |         ^~~~~~
shainc.c:1:1: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’
  +++ |+#include <stdio.h>
    1 | int main() {
shainc.c:2:9: error: incompatible implicit declaration of built-in function ‘printf’ [-Werror=builtin-declaration-mismatch]
    2 |         printf("hi\n");
      |         ^~~~~~
shainc.c:2:9: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’
cc1: all warnings being treated as errors
$ echo $?
2

Makefile nonzero

  • Makefile returns the error value:
$ cat Makefile
build:
        gcc shainc.c -Wall -Wextra -Werror -Wpedantic -O2 -o shainc
$ make 2>/dev/null ; echo $?
gcc shainc.c -Wall -Wextra -Werror -Wpedantic -O2 -o shainc
2

Actions

  • We can use make and make check within GitHub actions:
ci/c-cpp.yml
    - name: make
      run: make
    - name: make check
      run: make check
  • And GitHub will report error status to us when we check in files.

End-to-end

Makefile

Makefile
check: build
    echo "15 characters." > 15char.txt
    echo "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum." > lipsum.txt
    curl https://github.com/cd-public/books/raw/main/pg1342.txt -o austen.txt 2>/dev/null
    # Makefile uses sh not bash so can't use <() and have to use the filesystem
    ./shainc 15char.txt > 15char.inc
    ./shainc lipsum.txt > lipsum.inc
    ./shainc austen.txt > austen.inc
    sha256sum 15char.txt > 15char.sum
    sha256sum lipsum.txt > lipsum.sum
    sha256sum austen.txt > austen.sum
    diff 15char.inc 15char.sum || diff lipsum.inc lipsum.sum || diff austen.inc austen.sum


build:
    gcc shainc.c -Wall -Wextra -Werror -Wpedantic -O2 -o shainc

clean:
    rm shainc *.txt *.inc *.sum

.yml

.github/workflows/c89.yml
name: C89 CI

on: push

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4
    - run: make

Outcomes

Todo

  • Get this working on your “shainc” or “enigma” or something.
  • Either in your main repo or in some other one.
  • Add “GitHub Actions CI/CD pipelines for DevOps” to your resume.