Category Archives: mozilla

bugzilla and performance, 2014

bugzilla is a large legacy perl application initially developed at the birth of the world wide web. it has been refactored significantly in the 15 years since its release; however, some design decisions made at the start of the project carry significant technical debt in today’s modern web.

while there have been a large number of micro-optimisations over the course of bugzilla’s life (generally as a result of performance profiling), there are limits to the benefits these sorts of optimisations can provide.

this year has seen a lot of focus on improving bugzilla’s performance within mozilla, centred around the time it takes to display a bug to authenticated users.

response-times-may-2013response-times-may-2014

tl;dr bugzilla is faster

memcached

the design of a modern large web application is generally centred around caches, between the business logic layer and the data sources, as well as between the users and the user interface. while bugzilla has been refactored to use objects, database abstraction, templates, etc, it had zero caching capabilities. this coupled with completely stateless processing of each request meant that every byte of html returned to the user was regenerated from scratch, starting with a new connection to the database.

towards the end of 2013 i worked on implementing a memcached framework into bugzilla [bug 237498].

retrofitting a caching mechanism into a large extendible framework proved to be a significant challenge. bugzilla provides the ability for developers to extend bugzilla’s functionality via extensions, including but not limited to adding new fields, user interface, or process. the extensions system conflicts with caching as it’s possible for an extension to conditionally alter an object in ways that would render it impossible to cache (eg. add a new field to an object only if the current user is a member of a particular group).

some compromises had to be made. instead of caching fully constructed objects, the cache sits between the object’s constructor and the database. we avoid a trip to the database, but still have to construct objects from that data (which allows extensions to modify the object during construction).

code which updated the database directly instead of using bugzilla’s objects had to be found and rewritten to use the objects or updated to manually clear the cache entries. extra care had to be taken as returning stale data could silently result in data loss. to placate concerns that these errors would be impossible to detect, the caller of an object’s constructor must pass in a parameter to “opt-in” to caching.

in 2014 i built upon the memcached framework to support most of bugzilla’s objects [bug 956233], with the “bugs” object being the only notable exception. memcached also caches bugzilla’s configuration parameters (classifications, products, components, groups, flags, …) [bug 987032]. although caching the “bugs” object would be beneficial given its central role in all things bugzilla, it is highly likely that enabling this by default would break bugzilla extensions and customisations as a high proportion of them update the database directly instead of using bugzilla’s object layer. this would manifest as data which is silently stale, making undetectable dataloss a high probability.

memcached support will be released with bugzilla 5.0, but has been live on bugzilla.mozilla.org (bmo) since february 2014.

instrumentation

while profilling tools such as nytprof have often been pointed at bugzilla, bmo’s high number of concurrent requests and usage patterns time and time again resulted in performance optimisations performing worse than expected once deployed to production.

we took the decision to deploy instrumentation code into bugzilla itself, reporting on each http request, database query, and template execution. as bugzilla is written in perl, support was absent for off-the-shelf instrumentation tools such as new relic, so we had to roll our own data collection and reporting system [bug 956230].

the collector wraps specific Bugzilla::DB, Bugzilla::Memcached and Bugzilla::Template calls via subclassing, then reports data to an elasticsearch cluster. currently all reporting solutions are ad-hoc and involve running of scripts which identify the most costly database queries and templates.

this data identified areas of bugzilla which require optimisation or caching. examples include the optimisation of a single query which shaved 200ms (~15%) off page load times for most users [bug 993894], caching of all queries which read an entire table [part of bug 987032], and caching of user groups [bugs 993939] and settings [bug 993926].

we continue to revisit the instrumentation reports in order to guide further improvements.

stylesheet concatenation and minification

improving the performance of a web applications isn’t limited to focusing on server-side execution speed.

due to bugzilla’s extensions we ended up in a situation where bugzilla was serving multiple small css files – on bmo we loaded 17 stylesheets as part of show_bug in comparison with 5 for an installation of bugzilla without any extensions installed.

similar to the issue encountered with memcached, extensions have complete control with regards to optionally loading stylesheets, which means any css concatenation and minification solution needed to be implemented at run-time.

[bug 977969] does exactly that – the template passes an array of stylesheets to load to bugzilla’s global header, where a hash of the array is used to find a unified stylesheet. simple minification is performed which dropped the stylesheet size from 54kb to 43kb on show_bug on bmo.

stylesheet concatenation and minification support will be released with bugzilla 5.0, and has been live on bugzilla.mozilla.org since may 2014.

database

in order to address performance issues caused by bugzilla’s use of the myisam table type, in march our DBAs upgraded our database cluster to mysql version 5.6. this was the result of analysis by the DBAs into replication and stability issues around our myisam table.

as mysql 5.6 adds support for fulltext indexing to its innodb table type, bugzilla was able to switch away from myisam. this immediately fixed the database replication issues, and provided a noticeable performance boost to searches involving comments.

looking forward

the next large project is to update bugzilla so the bug object can use memcached on an unmodified installation without any non-default extensions enabled. for reasons previously covered it’s unlikely we’ll ship a version of bugzilla with this enabled by default, however this will allow sites to audit their own code (if any) and enable caching of the bug object if required.

we will build on the css concatenation and minification work to provide the same benefits to javascript files.

we periodically enable instrumentation and use it to identify the next set of targets for optimisation. this will continue for the foreseeable future.

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1029991] “bug_comments” hook should run after comment tags are pre-loaded
  • [1023865] New product (Tree Management) & components for Treeherder
  • [669535] User pref for “Possible Duplicates”
  • [1030617] editing a product’s groups clears the suggested reviewer list
  • [1030747] Recent changes to secure bugmail subject lines break GMail threading
  • [1028027] cloning a bug pre-fills mentors with “—” instead of an empty value
  • [1031428] Move the “Mentors” field under “QA Contact” field

discuss these changes on mozilla.tools.bmo.

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1027037] [bzapi] history.changer is not a full email address
  • [1027060] [bzapi] bug.id is a string if the response includes comments
  • [1012782] Expose cf_last_resolved in the RPC API
  • [1028377] Add Swag Requests wiki link to the Reps swag form
  • [1027638] Use product/component in subject of secure bugs rather than duplicating the bug number
  • [1027718] Bugzilla::Sentry fails with error if mod_perl is not installed on the system
  • [943146] Needinfo dropdown should include mentor

discuss these changes on mozilla.tools.bmo.

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1021218] merge-users.pl fails if both users have bug_user_last_visit entries for the same bug
  • [1017648] migrate mentors from the whiteboard to the mentor field
  • [1026421] “No QA contact” represented by “” instead of null
  • [1026415] Some empty fields are present as “” rather than being absent
  • [1026862] updating a bug via an api or bulk bug update clears the mentor field
  • [1026400] id, attachment.id, attachment.bug_id are returned as strings, not integers

discuss these changes on mozilla.tools.bmo.

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [849102] mydashboard should lazy-load the ‘last change’ information
  • [962424] jobqueue’s worker process should process messages in batches
  • [978146] activity entry when setting flags isn’t split across multiple rows
  • [1015994] attachment.cgi titles should include bug and/or attachment text
  • [960830] Update Brand Engagement Initiation Form
  • [1023633] Add a webservice_before_call Hook
  • [1022923] Add index to bug_user_last_visit.last_visit_ts
  • [956892] collapsed comments should display tags to indicate the reason for auto-collapsing
  • [880669] Extend current BzAPI BMO extension to contain compatibility changes on top of native rest
  • [649691] Add a “mentor” and “mentored bug type” field to bmo

after a herculean effort by dkl, this week’s update brings native bzapi support to bugzilla’s production infrastructure.  bzapi operates as a proxy server external to bugzilla, developed and operated by gerv. it is a reflection of gerv’s commitment and abilities that bzapi is by far the primary api which mozilians use to interface with bzapi. moving bzapi into bugzilla itself should improve performance and stability.

the plan with regards to migration is to continue to run the current bzapi proxy server until we’re happy with this code. at that point we’ll redirect bzapi calls made against the api-dev endpoint to the new endpoint.

please help us test this by changing your endpoints from https://api-dev.bugzilla.mozilla.org/latest/ to https://bugzilla.mozilla.org/bzapi/

eg.
old: https://api-dev.bugzilla.mozilla.org/latest/bug/880669
new: https://bugzilla.mozilla.org/bzapi/bug/880669

if you find any issues don’t hesitate to file bugs in the bzapi-compatibility component.

 

discuss these changes on mozilla.tools.bmo.

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1020425] Typo in “Updated since last visit” part of My Dashboard
  • [1018811] updated account name not reflected in “Reported” field of bugs
  • [1019747] New user story field in Tracking product is getting copied from Description, not User Story when cloning
  • [1020708] Copyright header missing from Bugzilla::BugUserLastVisit
  • [1016273] Bugzilla whine emails missing change to From header made by bug 1010751
  • [1020023] update a user’s last-visited timestamp for a bug when they perform any actions on it
  • [720832] Add the ability to watch component starting with a string
  • [1022500] cloning a firefox tracking flags throws “The component named ‘Template::Document=HASH(0x7f7ffacdb4c8)’ does not exist. “

discuss these changes on mozilla.tools.bmo.

bugzilla can now show bugs that have been updated since you last visited them

thanks to dylan‘s work on bug 489028, bugzilla now tracks when you view a bug, allowing you to search for bugs which have been updated since you last visited them.

on bugzilla.mozilla.org this has been added to “my dashboard“:

my dashboard - last visited

bugzilla will only track a bug if you are involved in it (you have to be the reporter, assignee, qa-contact, or on the cc list), and if you have javascript enabled.

the “last visit” field can be used as part of a custom search, and is most useful when used in conjunction with the “%last_changed%” pronoun (which expands to the bug’s last-changed timestamp).

last visited - custom search

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1003970] Tagging comments not shown in user activity
  • [993939] Bugzilla::User::Setting::groups() should use memcached
  • [1000917] Record last-visited time of bugs when logged in
  • [1016273] Bugzilla whine emails missing change to From header made by bug 1010751

discuss these changes on mozilla.tools.bmo.

 

firebot has a new owner

to echo wolf’s post – “The End of Firebot – A New Beginning”, i’m the new maintainer of the irc bot firebot.

firebot provided an excellent service, and i’m happy to ensure firebot’s continued existence.

i’d like to thank wolf for running firebot for a decade, and for allowing me to take over.

happy bmo push day!

the following changes have been pushed to bugzilla.mozilla.org:

  • [1013953] Update the researchers.html.tmpl page to link to mhoye automated sanitized database dumps
  • [993223] Notify Review Board when a bug is made confidential
  • [1003950] automatically disable accounts based on the number of comments tagged as “abusive”
  • [1014374] concatenate and slightly minify css files
  • [1013760] Add “secure mail” metadata to email headers
  • [1009216] Add link to a wiki page describing common whiteboard tags
  • [1015290] Fix typo on Reps Mentorship Form
  • [1003386] Create new “Mozilla Foundation Operations” product
  • [1013788] it’s possible to get bugzilla to redirect to any url by setting the content-type of an attachment after uploading it

discuss these changes on mozilla.tools.bmo.

Follow

Get every new post delivered to your Inbox.

Join 87 other followers