Missing expected images:
Cloud_base qcow2 x86_64 Atomic qcow2 x86_64 Cloud_base raw-xz x86_64 Xfce raw-xz armhfp Atomic raw-xz x86_64
Failed openQA tests: 21/107 (x86_64), 18/18 (i386), 1/2 (arm)
New failures (same test did not fail in Rawhide-20170119.n.0):
ID: 55308 Test: x86_64 Everything-boot-iso install_default URL: https://openqa.fedoraproject.org/tests/55308 ID: 55314 Test: x86_64 Workstation-live-iso base_services_start URL: https://openqa.fedoraproject.org/tests/55314 ID: 55317 Test: x86_64 Workstation-live-iso desktop_update_graphical URL: https://openqa.fedoraproject.org/tests/55317 ID: 55319 Test: x86_64 Workstation-live-iso desktop_browser URL: https://openqa.fedoraproject.org/tests/55319 ID: 55321 Test: x86_64 Workstation-live-iso desktop_notifications_postinstall URL: https://openqa.fedoraproject.org/tests/55321 ID: 55327 Test: i386 Workstation-boot-iso memory_check URL: https://openqa.fedoraproject.org/tests/55327 ID: 55337 Test: x86_64 KDE-live-iso desktop_browser URL: https://openqa.fedoraproject.org/tests/55337
Old failures (same test failed in Rawhide-20170119.n.0):
ID: 55290 Test: x86_64 Server-dvd-iso install_repository_nfs_variation URL: https://openqa.fedoraproject.org/tests/55290 ID: 55292 Test: x86_64 Server-dvd-iso server_role_deploy_domain_controller URL: https://openqa.fedoraproject.org/tests/55292 ID: 55294 Test: x86_64 Server-dvd-iso server_cockpit_default URL: https://openqa.fedoraproject.org/tests/55294 ID: 55298 Test: x86_64 Server-dvd-iso server_role_deploy_database_server URL: https://openqa.fedoraproject.org/tests/55298 ID: 55302 Test: x86_64 Server-dvd-iso base_services_start URL: https://openqa.fedoraproject.org/tests/55302 ID: 55306 Test: i386 Server-boot-iso install_default URL: https://openqa.fedoraproject.org/tests/55306 ID: 55307 Test: i386 Server-dvd-iso install_default URL: https://openqa.fedoraproject.org/tests/55307 ID: 55310 Test: i386 Everything-boot-iso install_default URL: https://openqa.fedoraproject.org/tests/55310 ID: 55322 Test: x86_64 Workstation-boot-iso install_default URL: https://openqa.fedoraproject.org/tests/55322 ID: 55325 Test: x86_64 Workstation-boot-iso install_default@uefi URL: https://openqa.fedoraproject.org/tests/55325 ID: 55326 Test: i386 Workstation-live-iso install_default URL: https://openqa.fedoraproject.org/tests/55326 ID: 55328 Test: i386 Workstation-boot-iso install_default URL: https://openqa.fedoraproject.org/tests/55328 ID: 55332 Test: x86_64 KDE-live-iso base_services_start URL: https://openqa.fedoraproject.org/tests/55332 ID: 55335 Test: x86_64 KDE-live-iso desktop_update_graphical URL: https://openqa.fedoraproject.org/tests/55335 ID: 55339 Test: x86_64 KDE-live-iso desktop_notifications_postinstall URL: https://openqa.fedoraproject.org/tests/55339 ID: 55340 Test: i386 KDE-live-iso install_default URL: https://openqa.fedoraproject.org/tests/55340 ID: 55341 Test: arm Minimal-raw_xz-raw.xz install_arm_image_deployment_upload URL: https://openqa.fedoraproject.org/tests/55341 ID: 55383 Test: x86_64 universal upgrade_kde_64bit URL: https://openqa.fedoraproject.org/tests/55383 ID: 55394 Test: x86_64 universal install_cyrillic_language URL: https://openqa.fedoraproject.org/tests/55394 ID: 55395 Test: x86_64 universal install_asian_language URL: https://openqa.fedoraproject.org/tests/55395 ID: 55400 Test: x86_64 universal install_rescue_encrypted@uefi URL: https://openqa.fedoraproject.org/tests/55400 ID: 55401 Test: i386 universal install_package_set_minimal URL: https://openqa.fedoraproject.org/tests/55401 ID: 55402 Test: i386 universal install_repository_http_graphical URL: https://openqa.fedoraproject.org/tests/55402 ID: 55403 Test: i386 universal install_scsi_updates_img URL: https://openqa.fedoraproject.org/tests/55403 ID: 55404 Test: i386 universal install_simple_encrypted URL: https://openqa.fedoraproject.org/tests/55404 ID: 55405 Test: i386 universal install_software_raid URL: https://openqa.fedoraproject.org/tests/55405 ID: 55406 Test: i386 universal install_btrfs URL: https://openqa.fedoraproject.org/tests/55406 ID: 55407 Test: i386 universal install_ext3 URL: https://openqa.fedoraproject.org/tests/55407 ID: 55408 Test: i386 universal install_lvmthin URL: https://openqa.fedoraproject.org/tests/55408 ID: 55409 Test: i386 universal upgrade_desktop_32bit URL: https://openqa.fedoraproject.org/tests/55409 ID: 55411 Test: i386 universal install_package_set_kde URL: https://openqa.fedoraproject.org/tests/55411 ID: 55413 Test: i386 universal upgrade_2_desktop_32bit URL: https://openqa.fedoraproject.org/tests/55413 ID: 55414 Test: x86_64 universal upgrade_2_kde_64bit URL: https://openqa.fedoraproject.org/tests/55414
Soft failed openQA tests: 52/107 (x86_64) (Tests completed, but using a workaround for a known bug)
New soft failures (same test did not soft fail in Rawhide-20170119.n.0):
ID: 55351 Test: x86_64 universal install_sata URL: https://openqa.fedoraproject.org/tests/55351 ID: 55352 Test: x86_64 universal install_sata@uefi URL: https://openqa.fedoraproject.org/tests/55352 ID: 55363 Test: x86_64 universal install_ext3 URL: https://openqa.fedoraproject.org/tests/55363 ID: 55364 Test: x86_64 universal install_xfs URL: https://openqa.fedoraproject.org/tests/55364 ID: 55366 Test: x86_64 universal install_no_swap URL: https://openqa.fedoraproject.org/tests/55366
Old soft failures (same test soft failed in Rawhide-20170119.n.0):
ID: 55285 Test: x86_64 Server-boot-iso install_default URL: https://openqa.fedoraproject.org/tests/55285 ID: 55286 Test: x86_64 Server-boot-iso install_default@uefi URL: https://openqa.fedoraproject.org/tests/55286 ID: 55287 Test: x86_64 Server-dvd-iso install_default_upload URL: https://openqa.fedoraproject.org/tests/55287 ID: 55288 Test: x86_64 Server-dvd-iso install_default@uefi URL: https://openqa.fedoraproject.org/tests/55288 ID: 55291 Test: x86_64 Server-dvd-iso install_repository_nfs_graphical URL: https://openqa.fedoraproject.org/tests/55291 ID: 55300 Test: x86_64 Server-dvd-iso install_updates_nfs URL: https://openqa.fedoraproject.org/tests/55300 ID: 55309 Test: x86_64 Everything-boot-iso install_default@uefi URL: https://openqa.fedoraproject.org/tests/55309 ID: 55343 Test: x86_64 universal install_package_set_minimal URL: https://openqa.fedoraproject.org/tests/55343 ID: 55344 Test: x86_64 universal install_anaconda_text URL: https://openqa.fedoraproject.org/tests/55344 ID: 55347 Test: x86_64 universal install_repository_http_graphical URL: https://openqa.fedoraproject.org/tests/55347 ID: 55348 Test: x86_64 universal install_mirrorlist_graphical URL: https://openqa.fedoraproject.org/tests/55348 ID: 55349 Test: x86_64 universal install_delete_pata URL: https://openqa.fedoraproject.org/tests/55349 ID: 55350 Test: x86_64 universal install_delete_pata@uefi URL: https://openqa.fedoraproject.org/tests/55350 ID: 55353 Test: x86_64 universal install_kickstart_user_creation URL: https://openqa.fedoraproject.org/tests/55353 ID: 55354 Test: x86_64 universal install_scsi_updates_img URL: https://openqa.fedoraproject.org/tests/55354 ID: 55355 Test: x86_64 universal install_multi URL: https://openqa.fedoraproject.org/tests/55355 ID: 55356 Test: x86_64 universal install_multi@uefi URL: https://openqa.fedoraproject.org/tests/55356 ID: 55357 Test: x86_64 universal install_simple_encrypted URL: https://openqa.fedoraproject.org/tests/55357 ID: 55358 Test: x86_64 universal install_simple_free_space URL: https://openqa.fedoraproject.org/tests/55358 ID: 55359 Test: x86_64 universal install_multi_empty URL: https://openqa.fedoraproject.org/tests/55359 ID: 55360 Test: x86_64 universal install_software_raid URL: https://openqa.fedoraproject.org/tests/55360 ID: 55361 Test: x86_64 universal install_delete_partial URL: https://openqa.fedoraproject.org/tests/55361 ID: 55362 Test: x86_64 universal install_btrfs URL: https://openqa.fedoraproject.org/tests/55362 ID: 55365 Test: x86_64 universal install_lvmthin URL: https://openqa.fedoraproject.org/tests/55365 ID: 55367 Test: x86_64 universal install_iscsi URL: https://openqa.fedoraproject.org/tests/55367 ID: 55369 Test: x86_64 universal install_simple_encrypted@uefi URL: https://openqa.fedoraproject.org/tests/55369 ID: 55370 Test: x86_64 universal install_simple_free_space@uefi URL: https://openqa.fedoraproject.org/tests/55370 ID: 55371 Test: x86_64 universal install_multi_empty@uefi URL: https://openqa.fedoraproject.org/tests/55371 ID: 55372 Test: x86_64 universal install_software_raid@uefi URL: https://openqa.fedoraproject.org/tests/55372 ID: 55373 Test: x86_64 universal install_delete_partial@uefi URL: https://openqa.fedoraproject.org/tests/55373 ID: 55374 Test: x86_64 universal install_btrfs@uefi URL: https://openqa.fedoraproject.org/tests/55374 ID: 55375 Test: x86_64 universal install_ext3@uefi URL: https://openqa.fedoraproject.org/tests/55375 ID: 55376 Test: x86_64 universal install_xfs@uefi URL: https://openqa.fedoraproject.org/tests/55376 ID: 55377 Test: x86_64 universal install_lvmthin@uefi URL: https://openqa.fedoraproject.org/tests/55377 ID: 55378 Test: x86_64 universal install_no_swap@uefi URL: https://openqa.fedoraproject.org/tests/55378 ID: 55379 Test: x86_64 universal install_kickstart_hdd URL: https://openqa.fedoraproject.org/tests/55379 ID: 55380 Test: x86_64 universal upgrade_minimal_64bit URL: https://openqa.fedoraproject.org/tests/55380 ID: 55382 Test: x86_64 universal upgrade_server_64bit URL: https://openqa.fedoraproject.org/tests/55382 ID: 55385 Test: x86_64 universal upgrade_2_minimal_64bit URL: https://openqa.fedoraproject.org/tests/55385 ID: 55387 Test: x86_64 universal upgrade_2_server_64bit URL: https://openqa.fedoraproject.org/tests/55387 ID: 55390 Test: x86_64 universal install_updates_img_local URL: https://openqa.fedoraproject.org/tests/55390 ID: 55391 Test: x86_64 universal install_shrink_ext4 URL: https://openqa.fedoraproject.org/tests/55391 ID: 55392 Test: x86_64 universal install_shrink_ntfs URL: https://openqa.fedoraproject.org/tests/55392 ID: 55396 Test: x86_64 universal install_kickstart_firewall_disabled URL: https://openqa.fedoraproject.org/tests/55396 ID: 55397 Test: x86_64 universal install_kickstart_firewall_configured URL: https://openqa.fedoraproject.org/tests/55397 ID: 55398 Test: x86_64 universal install_kickstart_nfs URL: https://openqa.fedoraproject.org/tests/55398 ID: 55412 Test: x86_64 universal install_repository_http_variation URL: https://openqa.fedoraproject.org/tests/55412
Passed openQA tests: 29/107 (x86_64)
New passes (same test did not pass in Rawhide-20170119.n.0):
ID: 55303 Test: x86_64 Server-dvd-iso base_service_manipulation URL: https://openqa.fedoraproject.org/tests/55303 ID: 55305 Test: x86_64 Server-dvd-iso server_firewall_default URL: https://openqa.fedoraproject.org/tests/55305 ID: 55311 Test: x86_64 Workstation-live-iso install_default_upload URL: https://openqa.fedoraproject.org/tests/55311 ID: 55312 Test: x86_64 Workstation-live-iso install_default@uefi URL: https://openqa.fedoraproject.org/tests/55312 ID: 55313 Test: x86_64 Workstation-live-iso base_selinux URL: https://openqa.fedoraproject.org/tests/55313 ID: 55315 Test: x86_64 Workstation-live-iso base_service_manipulation URL: https://openqa.fedoraproject.org/tests/55315 ID: 55316 Test: x86_64 Workstation-live-iso base_update_cli URL: https://openqa.fedoraproject.org/tests/55316 ID: 55318 Test: x86_64 Workstation-live-iso desktop_terminal URL: https://openqa.fedoraproject.org/tests/55318 ID: 55320 Test: x86_64 Workstation-live-iso desktop_notifications_live URL: https://openqa.fedoraproject.org/tests/55320 ID: 55323 Test: x86_64 Workstation-boot-iso memory_check URL: https://openqa.fedoraproject.org/tests/55323 ID: 55324 Test: x86_64 Workstation-boot-iso memory_check@uefi URL: https://openqa.fedoraproject.org/tests/55324 ID: 55381 Test: x86_64 universal upgrade_desktop_64bit URL: https://openqa.fedoraproject.org/tests/55381 ID: 55384 Test: x86_64 universal upgrade_desktop_encrypted_64bit URL: https://openqa.fedoraproject.org/tests/55384 ID: 55386 Test: x86_64 universal upgrade_2_desktop_64bit URL: https://openqa.fedoraproject.org/tests/55386 ID: 55389 Test: x86_64 universal upgrade_2_desktop_encrypted_64bit URL: https://openqa.fedoraproject.org/tests/55389 ID: 55393 Test: x86_64 universal install_european_language URL: https://openqa.fedoraproject.org/tests/55393
Skipped openQA tests: 3 of 127
Installed system changes in test x86_64 Server-boot-iso install_default: 2 packages(s) added since previous compose: cockpit-system, libreport-plugin-systemd-journal 3 packages(s) removed since previous compose: cockpit-shell, compat-readline6, libssh Previous test data: https://openqa.fedoraproject.org/tests/55153#downloads Current test data: https://openqa.fedoraproject.org/tests/55285#downloads
Installed system changes in test x86_64 Server-boot-iso install_default@uefi: 2 packages(s) added since previous compose: cockpit-system, libreport-plugin-systemd-journal 3 packages(s) removed since previous compose: cockpit-shell, compat-readline6, libssh Previous test data: https://openqa.fedoraproject.org/tests/55154#downloads Current test data: https://openqa.fedoraproject.org/tests/55286#downloads
Installed system changes in test x86_64 Everything-boot-iso install_default@uefi: 1 packages(s) removed since previous compose: compat-readline6 System load changed from 0.09 to 0.23 Previous test data: https://openqa.fedoraproject.org/tests/55177#downloads Current test data: https://openqa.fedoraproject.org/tests/55309#downloads
Installed system changes in test x86_64 KDE-live-iso install_default_upload: 1 packages(s) added since previous compose: kdiagram System load changed from 0.52 to 1.01 Used mem changed from 372 MiB to 514 MiB Previous test data: https://openqa.fedoraproject.org/tests/55275#downloads Current test data: https://openqa.fedoraproject.org/tests/55329#downloads
Installed system changes in test x86_64 KDE-live-iso install_default@uefi: 1 packages(s) added since previous compose: kdiagram System load changed from 0.64 to 0.84 Used mem changed from 374 MiB to 516 MiB Previous test data: https://openqa.fedoraproject.org/tests/55195#downloads Current test data: https://openqa.fedoraproject.org/tests/55330#downloads
Installed system changes in test x86_64 universal install_package_set_kde: 1 packages(s) added since previous compose: kdiagram Previous test data: https://openqa.fedoraproject.org/tests/55231#downloads Current test data: https://openqa.fedoraproject.org/tests/55368#downloads
x86_64 Workstation-boot-iso memory_check peak memory: 717MiB x86_64 Workstation-boot-iso memory_check@uefi peak memory: 715MiB
On Fri, 2017-01-20 at 17:33 +0000, Fedora compose checker wrote:
x86_64 Workstation-boot-iso memory_check peak memory: 717MiB x86_64 Workstation-boot-iso memory_check@uefi peak memory: 715MiB
So this is another shiny new openQA / check-compose feature: there's now a test which just runs an install with the `debug` kernel param, which triggers anaconda to record memory usage throughout the install process, then uploads the usage data file.
check-compose then notices jobs which do this, grabs the memory usage data file, and includes the peak usage during installation in the report. Tomorrow, it should include a comparison to today's usage (we don't get the comparison today, because today's the first compose we have the data for). If the usage changes significantly and we want to know why, we can go check the raw data file for more information (I guess I should enhance check-compose to link to it).
For the moment we're just running this test on the Workstation boot ISO, as it's likely to be the most memory-intensive install in openQA's set (network installs use more memory than DVD installs, and the Workstation network install has more packages than the Server network install).
As for the test results, there is a whole bunch of stuff going on, too much to break out test-by-test. Here are the highlights:
1) There is some kind of systemd bug which essentially means that occasionally, the system boots to emergency mode instead of booting properly. This affects about a half dozen tests every day, on average - never the same ones, the bug is quasi-random, it just...happens sometimes. That's https://bugzilla.redhat.com/show_bug.cgi?id=1414904 . I'm currently trying to pin that one down.
2) There is an AVC that appears on boot on just about all installs of Rawhide at present: https://bugzilla.redhat.com/show_bug.cgi?id=1392161 this causes the large number of soft failures, because many openQA tests now run a check for any AVCs or crash notifications after installation, and if any are found, it's considered a soft failure.
3) All i686 installs (and live image boots...really, Rawhide is just entirely broken on i686) are broken and have been since (IIRC) systemd v232. I think someone pointed me to a bug which may be the cause of this, but I've lost the reference. It's on my list to figure out soon.
4) KDE upgrades are running into a dependency issue: 'kde-runtime- drkonqi-16.08.3-3.fc24.x86_64 requires kde-runtime = 16.08.3-3.fc24, but none of the providers can be installed'. I haven't reported this properly yet, sorry, I should.
5) Russian installs are failing because of an anaconda bug: https://bugzilla.redhat.com/show_bug.cgi?id=1413813
6) The desktop_browser test needs a needle update, I'll take care of that.
7) All the 'check if all services start correctly' tests for various images fail because of this SELinux/systemd issue: https://bugzilla.redhat.com/show_bug.cgi?id=1412750
It occurs to me that we have kind of a recurring story: 'new systemd release lands in Rawhide, stuff breaks because it does stuff SELinux doesn't expect it to'. It might be nice if we could organize some kind of co-ordination between systemd and SELinux folks such that the appropriate SELinux permissions get added *before* the new systemd release lands. Of course, it would make things easier if we could provide upstream systemd with a nice easy way to test their git master on Fedora Rawhide, I guess.