W3C Public Bugzilla shutdown

• Archival
• Bug lists
• Bug pages
• Dependency tree pages
• Excluded pages
• Open questions
• Migration to GitHub
• List of products
• Migration scripts
• Other instances

W3C discontinued its public bugzilla instance as of April 1 2019, as announced at the Nov 2018 AC meeting.

The contents of this service were archived and will continue to be served in place at the same URIs. Some groups also migrated their issues to github.

This page has details on our plans for both Archival and Migration to GitHub.

Archival

The archiving was done by crawling the public site as an anonymous user (not logged in): this means that users on the site are identified by name and/or login id but not email address. This was done to avoid keeping thousands of email addresses available in the archive in perpetuity, leading to increased spam.

A few details about pages in the test archive are listed below.

Bug lists

• only specific bug lists are crawled and archived: lists of all bugs for a given product, and lists of all bugs for a given product/component.
• columns included in bug lists are: product, component, assigned_to, bug_status, resolution, short_desc, opendate, changeddate.
• links to specific queries will no longer work as intended; instead a list of all bugs for that component will be shown. Each archived bug list page has a note about this at the top.
• links to reorder a bug list by id/assignee/status are replaced with client-side sorting using tablesorter (requires js)
• a CSV version of each bug list is crawled and archived, with a fixed set of columns: product, component, assigned_to, bug_status, resolution, short_desc, opendate, changeddate — this list is easy to change if there are other columns people would find useful.
• XML versions are not crawled and archived: this is doable, but would be a minor nuisance and it seems unnecessary given that XML versions of individual bug pages will be available.
• links to atom and ical versions of bug lists are removed.

Bug pages

• nav bars removed from top and bottom
• 'format for printing' and 'clone this bug' links removed
• XML version is crawled and archived
• history / activity logs are crawled and archived
• dependency trees are crawled and archived, including resolved bugs (default is to omit them)
• dependency graph links are removed (they were broken anyway)
• file attachments are crawled and archived, but not served with the correct mime type because that would be a minor nuisance and may pose security issues: for now, these are all served as application/octet-stream

Dependency tree pages

• 'view as bug list' links do not work when there is more than one bug (sample)
• resolved bugs are included on dependency tree pages; these had been omitted by default.

Excluded pages

Some URIs are explicitly excluded from the archived version, served 410 Gone responses:

• createaccount.cgi
• describekeywords.cgi
• enter_bug.cgi
• jsonrpc.cgi
• page.cgi
• query.cgi
• quips.cgi
• report.cgi
• request.cgi
• search_plugin.cgi
• xmlrpc.cgi

Open questions

(no longer pertinent since the archiving has been done)

• are there specific pages/views that are not available in the sample archive that should be?
• which columns should be included in bug lists by default?
• which columns should be included in the CSV versions of bug lists by default?
• do we need to crawl and archive the XML versions of bug lists? (see above)
• are there some XMLRPC pages that could usefully be crawled and archived, in case someone wants to use them to import bugs into github post-shutdown?
• should the crawl be done as a logged-in user, so we have full email addresses for users? If so, how should we handle the spam issue?

Migration to GitHub

(this section still needs work)

In addition to the archival noted above, we plan to help groups copy issues from bugzilla into github issues, if desired.

List of products

W3C's public bugzilla instance has 30000+ bugs organized into 63 products, with 484 total components.

Summary of products in W3C's public bugzilla instance: (a work in progress)

Migration Scripts

Potential scripts to migrate bugs from bugzilla into github issues:

@berestovskyy bugzilla2github

This option seems most promising from what I have seen; here is a detailed writeup based on a forked version that includes lots of implementation details incl github API rate limits, etc. At least some of the interesting changes in the fork seem to have been merged upstream. (Théo Zimmermann is acked on @berestovskyy's page)

Interesting notes from the author of the detailed writeup above, included in this detailed analysis of switching bug trackers:

The only complicated part of the bug tracker switch was the migration of preexisting bug reports. We reused a tool by Andriy Berestovskyy which is designed to import Bugzilla reports (extracted as an XML dump) to GitHub using its REST API. The bugs are imported in an order designed to preserve numbers whenever possible. Bugs whose number is unavailable (e.g. because the number is already taken by a GitHub pull request) are postponed and renumbered. We implemented several improvements to make the tool better fit our needs:

• Allowing non-consecutive bug numbers: Our imported set of bug reports had some holes in the numbering due to deleted bugs. We use postponed bugs to fill the holes. This improvement has now been integrated upstream.
• Saving a table of correspondence for renumbered bugs: This was used later to redirect the old bug URLs to the new ones.
• Using the GitHub issue import beta API and overcoming the GitHub rate limits: Creating a new issue or a new comment through the normal GitHub REST API will trigger notifications (for people who are watching the repository or are mentioned in the issue thread). Such actions are therefore subject to a strict rate limit which prevented using this tool for importing more than a few hundred bugs. Fortunately, GitHub provides a (beta) issue import API which, in addition to not triggering notifications, also allows importing one bug, its comments, and meta information such as closed status and assignee in a single request (thus reducing the duration to import 4900 bugs to just a few hours). Finally, using this API allows to keep the dates of imported bug reports and comments, which is very useful to this study. We didn't manage to import correct closing dates, so we won't study the impact of the bug tracker switch on the time it takes to close a bug.

@jussi-kalliokoski

Discussion on Stack Overflow; Code. Seems to have been used by the Audio WG to migrate their issues. (or some of them)

Sample issue: Bugzilla, Github.

@dill BugzillaMigrate

Code

@heyatoito

Notes on migration of @w3c/webcomponents issues: (not sure about impl details)

Sample issue: Bugzilla, Github.

Other instances

W3C's Member and Team instances of bugzilla were shut down in Mar 2014. They had a relatively small number of bugs (950 and 270), compared to 30000+ in the public instance. For those instances we did a very simple crawl and archive of individual bug pages as well a few overview pages and lists of products and components. Authenticated uses can view the archived Member instance or Team instance.