Software Engineering at Google

1. Automated testing is foundational for software evolution and scalability

The adoption of developer-driven automated testing has been one of the most transformational software engineering practices at Google.

Prevention is key. Automated testing prevents bugs from escaping into production and affecting users. It also supports the ability to change software with confidence, enabling faster iteration and adaptation to evolving technologies and market conditions. At Google's scale, even a small percentage of test failures can waste enormous amounts of engineering time, making automation crucial.

Continuous improvement. Automated tests provide immediate feedback to developers, allowing them to catch and fix issues early in the development process. This rapid feedback loop accelerates development and improves overall code quality. Moreover, automated tests serve as living documentation, helping engineers understand how the system is intended to work and providing examples of proper usage.

Benefits of automated testing:

• Catches bugs early
• Enables confident changes
• Supports faster iteration
• Provides living documentation
• Scales with codebase growth

2. Effective testing balances different types and scopes of tests

At Google, we aim for a mix of about 80% unit tests and 20% broader-scoped tests.

Test pyramid strategy. Google employs a test pyramid approach, with a foundation of numerous small, fast unit tests, complemented by fewer integration tests and even fewer end-to-end tests. This strategy balances the need for quick feedback with comprehensive system validation.

Size and scope considerations. Tests are classified by size (small, medium, large) and scope (unit, integration, end-to-end). Small tests run in a single process and are fast and deterministic. Medium tests can span multiple processes but are limited to a single machine. Large tests can span multiple machines and are used for full system testing. By carefully considering the appropriate size and scope for each test, teams can optimize for both speed and thoroughness.

Test classification:

• Size: Small, Medium, Large
• Scope: Unit, Integration, End-to-End
  Benefits of balanced testing:
• Fast feedback from unit tests
• Comprehensive coverage from integration and end-to-end tests
• Efficient use of testing resources

3. Test maintainability is crucial for long-term productivity

Maintainable tests are ones that "just work": after writing them, engineers don't need to think about them again until they fail, and those failures indicate real bugs with clear causes.

Avoiding brittle tests. Brittle tests break in response to unrelated changes, draining productivity and morale. To prevent this, tests should be written against public APIs rather than implementation details. This approach ensures that tests fail only when there are meaningful changes to the system's behavior.

Clear and focused tests. When tests do fail, it's crucial that the cause is immediately apparent. Clear tests have a single, well-defined purpose and avoid testing multiple behaviors simultaneously. They should be complete (containing all necessary information) and concise (free from irrelevant details).

Characteristics of maintainable tests:

• Focused on public APIs
• Clear purpose
• Complete information
• Concise implementation
  Benefits:
• Reduced maintenance overhead
• Easier debugging
• Improved developer productivity

4. Write tests against public APIs to reduce brittleness

By far the most important way to ensure that tests don't need to change unless the requirements of the system being tested change is to write tests that invoke the system being tested in the same way its users would.

Public API focus. Testing through public APIs ensures that tests validate the system's behavior from a user's perspective. This approach makes tests more resilient to internal refactoring and changes, as long as the public contract remains the same.

Defining public APIs. The concept of a "public API" can vary depending on the unit being tested. It may include methods, classes, or entire packages that are intended for use by other parts of the system. The key is to identify the appropriate level of abstraction for each unit and test at that level.

Benefits of testing public APIs:

• Reduced test brittleness
• Better alignment with user expectations
• Easier refactoring of internal implementation
  Considerations for identifying public APIs:
• Intended usage by other parts of the system
• Level of abstraction appropriate for the unit
• Long-term stability of the interface

5. Strive for unchanging tests to minimize maintenance overhead

The ideal test is unchanging: after it's written, it never needs to change unless the requirements of the system under test change.

Types of code changes. Understanding the types of changes that occur in a codebase helps in writing stable tests. These include pure refactorings, new features, bug fixes, and behavior changes. Ideally, only behavior changes should require updates to existing tests.

Designing for stability. To achieve unchanging tests, focus on testing behaviors rather than implementation details. Use test doubles judiciously, preferring real objects when possible to avoid over-specifying interactions. State testing (verifying the end state) is generally more stable than interaction testing (verifying specific method calls).

Strategies for unchanging tests:

• Test behaviors, not implementation
• Use state testing over interaction testing
• Minimize use of test doubles
  Benefits:
• Reduced maintenance cost
• Increased confidence in refactoring
• Improved test suite longevity

6. Focus on testing behaviors rather than methods

Rather than writing a test for each method, write a test for each behavior.

Behavior-driven testing. A behavior is a guarantee that a system makes about how it will respond to inputs in a given state. Framing tests around behaviors rather than methods leads to clearer, more focused tests that are easier to maintain and understand.

Expressing behaviors. Behaviors can often be expressed using "given-when-then" statements. This format clearly defines the initial state, the action taken, and the expected outcome. By structuring tests this way, engineers can more easily identify and test all relevant scenarios.

Components of a behavior-driven test:

• Given: Initial state
• When: Action taken
• Then: Expected outcome
  Advantages:
• Clearer test purpose
• Easier to identify missing test cases
• More resilient to implementation changes

7. Cultivate a strong testing culture within the organization

At Google, we've found that engineers sometimes need to be persuaded that testing via public APIs is better than testing against implementation details.

Cultural shift. Establishing a strong testing culture requires ongoing effort and leadership support. At Google, this shift involved initiatives like orientation classes for new hires, the Test Certified program to improve team testing practices, and the quirky but effective "Testing on the Toilet" initiative to spread testing knowledge.

Continuous reinforcement. Maintaining a testing culture requires constant reinforcement through code reviews, team discussions, and recognition of good testing practices. It's important to demonstrate the value of testing in terms of improved productivity, code quality, and reduced maintenance costs.

Strategies for building a testing culture:

• Education and training programs
• Recognition of good testing practices
• Integration of testing into the development workflow
  Benefits:
• Improved code quality
• Faster development cycles
• Reduced maintenance costs

8. Leverage automation and tooling to support testing efforts

We use tools like clang-tidy (for C++) and Error Prone (for Java) to automate the process of enforcing rules.

Automated enforcement. Tools that automatically check for common errors, style violations, and other issues can significantly reduce the burden on human reviewers and catch problems early in the development process. These tools can be integrated into the development environment and continuous integration pipelines.

Custom tooling. Google has developed custom tools like TAP (Test Automation Platform) to support its testing efforts at scale. These tools help manage the complexity of running millions of tests across thousands of changes daily.

Types of automation tools:

• Static analysis tools
• Linters
• Continuous integration systems
• Custom test runners
  Benefits of automation:
• Consistent enforcement of rules
• Early detection of issues
• Reduced cognitive load on developers and reviewers

9. Balance human judgment with automated testing for comprehensive quality assurance

Automated testing is not suitable for all testing tasks.

Complementary approaches. While automated testing forms the foundation of Google's quality assurance efforts, certain aspects of software quality still require human judgment. This includes evaluating user experience, detecting complex security vulnerabilities, and performing exploratory testing to uncover unexpected issues.

Targeted human involvement. By automating the majority of testing tasks, human testers can focus on areas where their creativity and intuition add the most value. This might include assessing the quality of search results, evaluating audio and video quality, or searching for complex security flaws.

Areas benefiting from human judgment:

• User experience evaluation
• Complex security testing
• Exploratory testing
• Qualitative assessments (e.g., audio/video quality)
  Best practices:
• Automate repetitive and well-defined tests
• Reserve human effort for high-value, judgment-intensive tasks
• Use exploratory testing to complement automated tests

Last updated: October 2, 2024