Testing Discussion

This session was a brainstorm to talk about what we want out of a testing system for the GNOME platform. We came up with a good list of what we wanted to accomplish as well as some thoughts for how to get there. We didn't create any concrete tasks in this session, but the afternoon session on Dogtail and (hopefully) the hackfest the next day got down to some actual approaches.

Goals

We want pervasive use of testing with high visibility, high availability, ease of use, and ease of integration. We want people to be able to write their own tests. We want to be able to uncover problems and regressions sooner than later. For testing, we want:

• Test AT-SPI infrastructure itself
• Test an AT-SPI implementation (e.g., GAIL)
• Test application functionality (via AT-SPI)
• Test assistive technology (consumer of AT-SPI)
• Test for compatibility (e.g., does new infrastructure break existing apps?)

Requirements

• Human/computer readable output
• Track failures back to source
• Test failure doesn't stop whole test system (Robust test harness; continue testing if a test fails; "-k feature like in make")
• Easy for people to run
• Easy for people to compose tests
• Dependency system: know which tests depend upon which other tests
• Tests should not require the entire build harness to work (e.g., application developer should be able to run their own tests standalone).
• Provide various ways to inform people of results/breakage/coverage (e-mail, IRC bot, web page)
• Open source

Technologies in Use Today

• LDTP
• Dogtail

• Rodney Dawe's a11y-test-suite: http://www.mail-archive.com/gnome-announce-list@gnome.org/msg03012.html

• Orca's test framework
• LSR's test framework
• FSG's ATK conformance test using AT-SPI

• LSB's GTK test suite for GTK & ATK & Pango and others [uses TET]

• lcov: http://ltp.sourceforge.net/coverage/lcov.php

KDE folks are also putting testing into the putback/tinderbox scheme.

When should a tests be run?

• Nightly [e.g., http://jhbuild.bxlug.be/builds/2006-10-05-0002/]

• Continuous
• On Change

Where should testing be done?

• Compilation
  • Do things even compile (e.g., handled by jhbuild today)?
• AT-SPI infrastructure functional tests
  • Can things connect to each other via AT-SPI?
  • Does base toolkit work (e.g., GTK widgets provide correct AT-SPI behavior)?
    • NOTE: should provide a GTK app or set of apps that exercise all GTK components (e.g., charge the GTK team to keep gtk-demo up-to-date with all widgets in GTK).
    • NOTE: want to make sure all 'unique' uses of the GTK API and things that have exposed bugs in GTK get rolled back into the GTK tests.
  • Application functional tests
    • Does the application work as expected?
    • What is the coverage of the tests (e.g., lcov info)?
    • Include additional AT-SPI tests for custom widgets
    • Tests should live in the CVS repository with the application
  • Assistive technology functional test
    • Does it provide the user experience it is supposed to provide?
    • May use application tests as input, but may also have other more specific tests

Ideas

• gnome-common "test" target: make it easy for individual app developers to run their own tests directly (does this force an
  • LDTP/Dogtail dependency)?

Issues

We need to encourage an active test writing community.

• Keeping test suite(s) up to date
  • New GTK widgets
  • New uses of existing widgets
  • Writing new tests when bugs are reported/fixed

Strategies

• Look at the live tinderbox work KDE is doing
• Integrate jhbuild with dogtail?
• Integrate with GARNOME?

• BuildBot

• Iago Toral Quiroga e-mail, 4Sept06 "About tests integration" to <build-brigade-list@gnome.org>

• Frysk tool at RedHat, a whole set of dogtail tests in their own internal build system on top of mawk (an RPM build)

  • No harness associated with dogtail; some logging capabilities; RHTS (RedHat Test System) is their harness, as part of a make check

  • Wrapped up as a 'make run'

• Talk with the BuildBrigade folks: <live.gnome.org/BuildBrigade>

Other Discussion

• JimGettys: modify jhbuild to use pkgconf info for dependencies

• JimGettys: want people to think about the following OLCP scenario: initial set of countries commit to deploying laptops; how do we integrate these new folks, resources into our community?

• JimGettys: only rational thing is that some a11y problem will be solved with different hardware than OLPC is providing

Action Items

• Need to look at jhbuild patches that propose test integration:

  • https://bugzilla.gnome.org/show_bug.cgi?id=318531

  • https://bugzilla.gnome.org/show_bug.cgi?id=349114

  • https://bugzilla.gnome.org/show_bug.cgi?id=308352

• Need to talk with BuildBrigade folks - they may have already solved a large part of the harness/integration problem for us.