happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1160929] add support for storing attachments in s3
  • [1190693] Rewrite auth delegation to use a server-side POST instead of a client-side GET to delegate API Key
  • [1184332] Add Restricted API calls for MozReview
  • [1190029] Would like a way to see which groups are ‘secure’ groups in editusers.cgi when diagnosing password reset issues
  • [1193590] Warning about obsolete patches on an unassigned bug
  • [1036872] Bugmail filtering allows through bugmail that should be blocked when combined with the standard email prefs
  • [1192854] change tabs on user preferences from horizontal to vertical layout
  • [1193878] Make a few more fields optional on the intern request form
  • [1192688] add ip to last used api key information

discuss these changes on mozilla.tools.bmo.

changing the default platform and operating-system on bugzilla.mozilla.org to all / all

bugzilla has a set of fields, “hardware” and “operating system”, that i’ll collectively call “platform” in this post.  their default values are detected from the reporter’s user-agent string when a bug is created.

unfortunately on bmo, the platform fields have two distinctly different meanings: the reporter’s platform and the platform a bug applies to. for too long have these two conflicting meanings coexisted within the same field, leading to confusion and a field that on many bugs is wrong or useless.

thanks to bug 579089 we plan on making the following changes early next week:

  • each product gains the ability to set their default platform
  • the default platform for all products initially will be all / all
  • a “use my platform” action will be added to enter-bug, allowing the bug reporter to quickly change from the product’s default
  • a “from reporter” button will be visible when viewing untriaged bugs, which sets the platform to the reporter’s

changing the face of bugzilla.mozilla.org


bugzilla.mozilla.org presents some interesting problems when it comes to UX; while it has a singular view of bug data, most teams use bugzilla in slightly different ways.  some of these differences surface in fields used to track work (keywords, flags, whiteboard, summary), how bugs are prioritised (whiteboard, custom fields, priority fields), and even how inter-bug dependancies are tracked.

the net result is trying to design an user interface for viewing bugs optimised for how people use bugzilla is a near impossible task.  in light of this, we’ve been working on a framework which allows us to deploy multiple experimental alternative view of bugs.


the goals are to enable alternative bug views in a way which enables rapid development and doesn’t force incomplete or broken implementations upon users.

implemented as a bugzilla extension, alternativeUI provides a harness for alternative bug views, and implements the view selection – currently as a simple user preference.  in the future we expect bugzilla to be able to automatically change view depending on the bug or your relationship to it.  for example, we could present a slightly (or radically!) different view of a bug to the assignee vs. the reviewer vs. someone not involved in the bug.

generic show_bug alternative

of course a framework for alternative bug views would be useless without an actual alternative.  i’ve been working on one which will likely be the basis of future experiments.  show/edit modality is at the core of this design — a bug is loaded with most fields as static text and switching to “edit mode” is required to make changes.

ideas we’re throwing around:

hiding fields

if a field doesn’t have a value set there’s no need for that field to be displayed by default.  removing those fields from the initial display greatly reduces the noise and complexity generally associated with bugzilla.

we can also use the user’s group membership or involvement in the bug as a cue to which data should be initially visible.  an example would be hiding the QE flags from users who are not associated with the bug or members of a the QE team.


when a bug is loaded right now in bugzilla, it has to load all the alternatives for many fields (products, components, versions, milestones, flags, …).  in most cases bugzilla’s fine-grained security model results in a measurable cost to generating these alternatives — you can see the difference yourself by logging out of bugzilla or opening a bug in a private tab and comparing the page load times against a logged-in request.

by loading a bug as read-only initially, we can defer loading the alternatives until after the user clicks on ‘edit’.

selected “editable by default”

requiring a mode switch to perform any change to a bug isn’t an ideal situation either – operations that are frequently performed should be easy to do with minimal extraneous steps.

the current implementation allows you to comment, CC, or vote for a bug without switching to edit mode.  this can be extended to, for example, allow the bug’s assignee to change a bug’s status/resolution without needing to switch to edit mode.




bugzilla and performance, 2014

bugzilla is a large legacy perl application initially developed at the birth of the world wide web. it has been refactored significantly in the 15 years since its release; however, some design decisions made at the start of the project carry significant technical debt in today’s modern web.

while there have been a large number of micro-optimisations over the course of bugzilla’s life (generally as a result of performance profiling), there are limits to the benefits these sorts of optimisations can provide.

this year has seen a lot of focus on improving bugzilla’s performance within mozilla, centred around the time it takes to display a bug to authenticated users.


tl;dr bugzilla is faster


the design of a modern large web application is generally centred around caches, between the business logic layer and the data sources, as well as between the users and the user interface. while bugzilla has been refactored to use objects, database abstraction, templates, etc, it had zero caching capabilities. this coupled with completely stateless processing of each request meant that every byte of html returned to the user was regenerated from scratch, starting with a new connection to the database.

towards the end of 2013 i worked on implementing a memcached framework into bugzilla [bug 237498].

retrofitting a caching mechanism into a large extendible framework proved to be a significant challenge. bugzilla provides the ability for developers to extend bugzilla’s functionality via extensions, including but not limited to adding new fields, user interface, or process. the extensions system conflicts with caching as it’s possible for an extension to conditionally alter an object in ways that would render it impossible to cache (eg. add a new field to an object only if the current user is a member of a particular group).

some compromises had to be made. instead of caching fully constructed objects, the cache sits between the object’s constructor and the database. we avoid a trip to the database, but still have to construct objects from that data (which allows extensions to modify the object during construction).

code which updated the database directly instead of using bugzilla’s objects had to be found and rewritten to use the objects or updated to manually clear the cache entries. extra care had to be taken as returning stale data could silently result in data loss. to placate concerns that these errors would be impossible to detect, the caller of an object’s constructor must pass in a parameter to “opt-in” to caching.

in 2014 i built upon the memcached framework to support most of bugzilla’s objects [bug 956233], with the “bugs” object being the only notable exception. memcached also caches bugzilla’s configuration parameters (classifications, products, components, groups, flags, …) [bug 987032]. although caching the “bugs” object would be beneficial given its central role in all things bugzilla, it is highly likely that enabling this by default would break bugzilla extensions and customisations as a high proportion of them update the database directly instead of using bugzilla’s object layer. this would manifest as data which is silently stale, making undetectable dataloss a high probability.

memcached support will be released with bugzilla 5.0, but has been live on bugzilla.mozilla.org (bmo) since february 2014.


while profilling tools such as nytprof have often been pointed at bugzilla, bmo’s high number of concurrent requests and usage patterns time and time again resulted in performance optimisations performing worse than expected once deployed to production.

we took the decision to deploy instrumentation code into bugzilla itself, reporting on each http request, database query, and template execution. as bugzilla is written in perl, support was absent for off-the-shelf instrumentation tools such as new relic, so we had to roll our own data collection and reporting system [bug 956230].

the collector wraps specific Bugzilla::DB, Bugzilla::Memcached and Bugzilla::Template calls via subclassing, then reports data to an elasticsearch cluster. currently all reporting solutions are ad-hoc and involve running of scripts which identify the most costly database queries and templates.

this data identified areas of bugzilla which require optimisation or caching. examples include the optimisation of a single query which shaved 200ms (~15%) off page load times for most users [bug 993894], caching of all queries which read an entire table [part of bug 987032], and caching of user groups [bugs 993939] and settings [bug 993926].

we continue to revisit the instrumentation reports in order to guide further improvements.

stylesheet concatenation and minification

improving the performance of a web applications isn’t limited to focusing on server-side execution speed.

due to bugzilla’s extensions we ended up in a situation where bugzilla was serving multiple small css files – on bmo we loaded 17 stylesheets as part of show_bug in comparison with 5 for an installation of bugzilla without any extensions installed.

similar to the issue encountered with memcached, extensions have complete control with regards to optionally loading stylesheets, which means any css concatenation and minification solution needed to be implemented at run-time.

[bug 977969] does exactly that – the template passes an array of stylesheets to load to bugzilla’s global header, where a hash of the array is used to find a unified stylesheet. simple minification is performed which dropped the stylesheet size from 54kb to 43kb on show_bug on bmo.

stylesheet concatenation and minification support will be released with bugzilla 5.0, and has been live on bugzilla.mozilla.org since may 2014.


in order to address performance issues caused by bugzilla’s use of the myisam table type, in march our DBAs upgraded our database cluster to mysql version 5.6. this was the result of analysis by the DBAs into replication and stability issues around our myisam table.

as mysql 5.6 adds support for fulltext indexing to its innodb table type, bugzilla was able to switch away from myisam. this immediately fixed the database replication issues, and provided a noticeable performance boost to searches involving comments.

looking forward

the next large project is to update bugzilla so the bug object can use memcached on an unmodified installation without any non-default extensions enabled. for reasons previously covered it’s unlikely we’ll ship a version of bugzilla with this enabled by default, however this will allow sites to audit their own code (if any) and enable caching of the bug object if required.

we will build on the css concatenation and minification work to provide the same benefits to javascript files.

we periodically enable instrumentation and use it to identify the next set of targets for optimisation. this will continue for the foreseeable future.

a tale of bugzilla performance

a high-level goal across multiple teams this year is to improve bugzilla.mozilla.org’s performance, specifically focusing on the time it takes to load a bug (show_bug.cgi).

towards this end, in q1 2014 i focused primarily on two things: implementing a framework for bugzilla to use memcached, and deep instrumentation of bmo in our production environment to help identify areas which require optimisation and could leverage memcached.

i’ll talk more about memcached in a later blog post.  today i’ll talk about a single little query.

the data gathered quickly identified a single query used to determine a user’s group membership was by far the slowest query, averaging more than 200 ms to complete, and was executed on every page:

  FROM groups, 
 WHERE user_group_map.user_id = 3881 
       AND (
           (user_group_map.isbless = 1 AND groups.id=user_group_map.group_id)
           OR (groups.id = group_group_map.grantor_id
               AND group_group_map.grant_type = 1
               AND group_group_map.member_id IN (20,19,10,9,94,23,49,2,119,..))

in bug 993894 i rewrote this query to:

  FROM user_group_map
 WHERE user_id = 3881
       AND isbless = 1
  FROM group_group_map
 WHERE grant_type = 1
       AND member_id IN (20,19,10,9,94,23,49,2,119,..)

which executes almost instantly.

the yellow bar on the following graph shows when this fix was deployed to bugzilla.mozilla.org:


bugzilla 4.2 – what’s new – searching

a major change between bugzilla 4.0 and bugzilla 4.2 comes in the form of changes to searching. the searching implementation in 4.2 was rewritten from scratch, removing some seriously convoluted code and adding a large amount of test cases.

notable differences include:

search results limiting

search results are initially limited to 500 bugs, with an additional click required to show up to 10,000 bugs.  it is no longer possible to generate a query which results in massive bug lists.

searching via APIs remains unlimited.

changes to relative dates

when using relative dates and times, -1w is now a synonym for -7d and means exactly 7 days.

previously, -1w meant the beginning of the week, which was confusing some users. the same confusion existed for -1d which was different from -24h, and for -1m which was different from -30d. now if you really want the beginning of the day, week or month, you must use -1ds, -1ws, and -1ms respectively, where “s” means “start of”.

this change will affect existing saved searches using relative dates.

improved custom search builder

the custom search builder on the advanced search tab has been improved; it’s now easier and quicker to construct queries.

predictable results

complex queries built by the custom search interface are easier to build and have more sensible results, as they are built using a more intuitive logic.