Software Engineering §

The data model §

Git stores four object types in a content-addressable store: blobs (file contents), trees (directory listings), commits (snapshots), and tags (named references). Each object is identified by a content-addressable hash (SHA-1 for existing repositories; Git 3.0 transitions new repositories to SHA-256). A commit points to a tree (the file system at that point) and one or more parent commits.

Example: You edit main.c and run git commit. Git creates a new blob for the file content, a new tree referencing that blob, and a new commit pointing to the tree and the previous commit. The branch pointer main advances to the new commit.

Branching strategies §

• Git Flow: main (production), develop (integration), feature/*, release/*, hotfix/*. Suited to release-driven projects.
• Trunk-based: everyone commits to main; feature flags control rollout. Short-lived branches ($< 1$ day).
• GitHub Flow: main + short-lived feature branches; pull requests for review.

Example: In a trunk-based team, Alice creates branch feat/login, commits three times, opens a pull request, and merges within a day. The branch is deleted. No long-lived integration branch exists.

Merge vs. rebase §

Merge preserves history (a merge commit shows the divergence); rebase creates a linear history (rewrites commit hashes). Rebase is preferred for local commits; merge for shared branches.

What a review covers §

A review examines: correctness (does the code do what the spec says?), readability (can another engineer follow it?), test coverage (are the changes tested?), and security (no injection, no secret leakage, no privilege escalation). See Security for the specific threat classes a reviewer should check.

Example: A pull request adds a new API endpoint. The reviewer checks: (1) input validation (no SQL injection), (2) error handling (no stack traces leaked to the client), (3) tests (a unit test for the happy path and one for a malformed input), (4) naming (the function name matches its behavior).

Review metrics §

• Review time: median time from PR opened to first review. Target: $< 4$ hours for small PRs.
• Defect escape rate: fraction of bugs found in production that should have been caught in review.
• PR size: median lines changed per PR. Smaller PRs ($< 400$ lines) are reviewed more thoroughly.

Practical guidelines §

• Review within one business day; stale reviews lose context.
• Keep PRs small: one logical change per PR.
• Use checklists for recurring concerns (error handling, logging, API compatibility).
• Automate what can be automated: linters, formatters, and static analysis run in CI before a human reviews.

The test pyramid §

The test pyramid organizes tests by cost and coverage. Many unit tests (fast, cheap, isolate logic), fewer integration tests (slower, test component interaction), few acceptance tests (slowest, test the full user journey). An inverted pyramid (many E2E tests) is fragile and slow.

Example: A payment service has 200 unit tests (each testing one function: calculateTax, validateCard, formatInvoice), 30 integration tests (testing the payment flow: card → gateway → ledger), and 5 acceptance tests (end-to-end: user clicks “Pay” → confirmation email arrives).

Property-based testing §

Instead of specific inputs, specify properties that must hold for all inputs. for all lists l: sort(sort(l)) == sort(l). Tools: QuickCheck (Haskell), Hypothesis (Python), proptest (Rust).

Example: You write a function merge(a, b) that merges two sorted lists. A property test asserts: for all sorted lists a, b: is_sorted(merge(a, b)) and for all sorted lists a, b: set(merge(a, b)) == set(a) ∪ set(b). The test framework generates hundreds of random inputs and checks the property holds for each.

Fuzzing §

Coverage-guided fuzzing (AFL, libFuzzer) uses coverage feedback to guide input generation. Structure-aware fuzzing (Jazzer, honggfuzz) uses the program’s type information to generate valid inputs.

Example: You fuzz a JSON parser. The fuzzer starts with {"key": "value"}, mutates bytes, and tracks which code paths are hit. When a new branch is reached (e.g., a nested object), the fuzzer focuses mutations there. After 10 million iterations, it finds a 3-byte input that triggers a buffer overflow.

Mutation testing §

Introduce small bugs (mutations) into the code; if the test suite does not catch the mutation, the tests are insufficient.

Example: The line if (x > 0) is mutated to if (x >= 0). If no test fails, the suite does not distinguish between the two, meaning the boundary case $x = 0$ is untested.

Authoritative sources

• xUnit Test Patterns (Meszaros)

Pipeline stages §

A typical pipeline: build → unit test → integration test → security scan → deploy to staging → acceptance test → deploy to production. The security scan stage (SAST, dependency audit, container image scan) is described in Security.

Example: A developer pushes a commit. The CI runner compiles the code (build), runs 200 unit tests (30 s), runs 30 integration tests against a test database (2 min), scans dependencies for known CVEs (1 min), and deploys to a staging environment. If the integration tests pass, the staging deployment triggers 5 acceptance tests. All green → the artifact is tagged v2.14.3 and is ready for production.

Deployment strategies §

• Blue-green: two identical environments; switch traffic from blue to green. Instant rollback: switch back.
• Canary: deploy to a small percentage of users (e.g., 5%); monitor metrics; roll out to 100% or roll back.
• Rolling update: replace instances one at a time (Kubernetes default). No extra capacity needed.

Example: A canary deployment of a new search algorithm: 5% of traffic (≈ 500 requests/min) hits the new version. After 10 minutes, the error rate is 0.2% (baseline 0.1%) and p99 latency is 120 ms (baseline 110 ms). The team rolls out to 25%, then 50%, then 100% over the next hour. If at any step the error rate exceeds 1%, the canary is rolled back automatically.

Infrastructure as Code §

Define infrastructure (servers, networks, databases) in code (Terraform, Pulumi, CloudFormation). Version-controlled, reviewable, reproducible. The target Operating System and Computer Hardware are declared in the IaC configuration, so the environment is identical from staging to production.

Example: A Terraform file declares: 3 t3.medium instances in an auto-scaling group, an ALB in front, an RDS PostgreSQL instance, and a VPC with public/private subnets. Running terraform apply creates or updates all of these. The same file is used for staging (1 instance) and production (3 instances) via a variable.

DORA metrics in practice §

The four DORA metrics are computed from CI/CD and observability data:

• Deployment Frequency: how often code reaches production (per day, per week).
• Lead Time for Changes: time from commit to production.
• Change Failure Rate: fraction of deployments that cause an incident.
• Failed Deployment Recovery Time: time to restore service after a failed deployment.

Teams are categorized as Elite, High, Medium, or Low performers based on these four metrics. The goal is not a specific number but continuous improvement: reduce lead time, increase deployment frequency, and keep failure rate and recovery time low.

The three pillars §

Logs (what happened), metrics (how much), traces (how it flowed). Together, they answer: “Is the system healthy? What is wrong? Where did it fail?”

Example: A user reports a slow checkout. The metrics dashboard shows p99 latency spiked at 14:02. The trace for that request shows: api-gateway (5 ms) → auth-service (12 ms) → inventory-service (1800 ms) → database (1750 ms). The logs for inventory-service at 14:02 show: WARN: connection pool exhausted, waiting 1750 ms for a connection. Root cause: the connection pool was too small for the traffic spike.

OpenTelemetry §

A vendor-neutral standard for instrumentation. The SDK collects spans, metrics, and logs; the collector exports them to a backend (Jaeger, Prometheus, Loki). The specification defines the data model (spans, attributes, resource) and the protocol (OTLP over gRPC or HTTP).

Example: A Go service uses the OpenTelemetry SDK to create a span for each HTTP request. The span includes attributes: http.method, http.status_code, db.system=postgres. The OTLP collector batches spans and exports them to Jaeger. In the Jaeger UI, you search by service=inventory-service and http.status_code=500 to find all failing requests.

Dashboards and correlation §

Grafana (metrics), Kibana (logs), Jaeger (traces). Correlate by trace ID: a slow request in the dashboard links to the trace, which links to the logs. A single trace ID threads through all three systems.

Chaos engineering §

Intentionally inject failures (kill a node, add latency, drop packets) to verify that the system behaves as expected. This is a practical application of Fault Tolerance principles: you test the system’s resilience before a real failure occurs. Tools: Chaos Monkey (Netflix), Litmus (Kubernetes).

Example: Every Friday at 10:00, a chaos experiment kills one pod in the payment-service deployment. The expected behavior: the load balancer routes to the remaining pods, the killed pod is restarted by the orchestrator, and the error rate stays below 0.1%. If the error rate spikes above 1%, the experiment is a failure and the team investigates before the next run.

Authoritative sources

• OpenTelemetry Specification

Problem: A team of 5 developers works on a web application. They deploy weekly, and each deployment takes 2 hours. A bug is found in production 3 days after deployment. How can they reduce the time-to-detection and time-to-recovery?

Git data model: objects (blobs, trees, commits, tags) stored in a content-addressable store (SHA-1 hash). A commit points to a tree (the file system at that point) and one or more parent commits.

Branching strategies:

• Git Flow: main (production), develop (integration), feature/*, release/*, hotfix/*.
• Trunk-based: everyone commits to main; feature flags control rollout. Short-lived branches ($< 1$ day).
• GitHub Flow: main + short-lived feature branches; PRs for review.

Merge vs. rebase: merge preserves history (a merge commit shows the divergence); rebase creates a linear history (rewrites commit hashes). Rebase is preferred for local commits; merge for shared branches.

Test pyramid: many unit tests (fast, cheap), fewer integration tests (slower), few acceptance tests (slowest). Inverted pyramid (many E2E tests) is fragile and slow.

Property-based testing: instead of specific inputs, specify properties that must hold for all inputs. for all lists l: sort(sort(l)) == sort(l). Tools: QuickCheck (Haskell), Hypothesis (Python), proptest (Rust).

Fuzzing: coverage-guided fuzzing (AFL, libFuzzer) uses coverage feedback to guide input generation. Structure-aware fuzzing (Jazzer, honggfuzz) uses the program’s type information to generate valid inputs.

Mutation testing: introduce small bugs (mutations) into the code; if the test suite does not catch the mutation, the tests are insufficient.

Pipeline stages: build → unit test → integration test → security scan → deploy to staging → acceptance test → deploy to production.

Deployment strategies:

• Blue-green: two identical environments; switch traffic from blue to green.
• Canary: deploy to a small percentage of users; monitor; roll out or roll back.
• Rolling update: replace instances one at a time (Kubernetes default).

Infrastructure as Code (IaC): define infrastructure (servers, networks, databases) in code (Terraform, Pulumi, CloudFormation). Version-controlled, reviewable, reproducible.

The three pillars: logs (what happened), metrics (how much), traces (how it flowed). Together, they answer: “Is the system healthy? What is wrong? Where did it fail?”

OpenTelemetry: a vendor-neutral standard for instrumentation. The SDK collects spans, metrics, and logs; the collector exports them to a backend (Jaeger, Prometheus, Loki).

Dashboards: Grafana (metrics), Kibana (logs), Jaeger (traces). Correlate by trace ID: a slow request in the dashboard links to the trace, which links to the logs.

Chaos engineering: intentionally inject failures (kill a node, add latency, drop packets) to verify that the system behaves as expected. Tools: Chaos Monkey (Netflix), Litmus (Kubernetes).

Reduce time-to-detection:

1. CI pipeline: every commit triggers build + unit tests + integration tests. Catches most bugs before they reach production.
2. Staging environment: deploy to a staging environment that mirrors production. Run acceptance tests and smoke tests.
3. Monitoring and alerting: set up metrics (error rate, latency, throughput) and alerts. A spike in 500 errors triggers an alert within minutes, not days.
4. Structured logging: log request IDs, user IDs, and error details. When an alert fires, the on-call engineer can find the relevant logs in seconds.

Reduce time-to-recovery:

1. Rollback: keep the previous deployment artifact. Roll back in minutes, not hours.
2. Feature flags: wrap risky features in flags. If a bug is found, disable the flag without redeploying.
3. Canary deployment: deploy to 5% of users first. If the error rate spikes, roll back the canary before it affects all users.
4. Blue-green: keep the old environment running. Switch traffic back if the new deployment fails.

Result: weekly 2-hour deployments become daily 10-minute deployments. Bugs are caught in CI (minutes) or in the canary (minutes), not in production (days).