Bug 13746 - auto-annotate bugzilla from git commits
Summary: auto-annotate bugzilla from git commits
Status: REOPENED
Alias: None
Product: glibc
Classification: Unclassified
Component: admin (show other bugs)
Version: unspecified
: P2 normal
Target Milestone: ---
Assignee: Carlos O'Donell
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-02-24 18:01 UTC by Roland McGrath
Modified: 2014-06-26 15:15 UTC (History)
4 users (show)

See Also:
Host:
Target:
Build:
Last reconfirmed:
fweimer: security-


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Roland McGrath 2012-02-24 18:01:00 UTC
We used to have this under CVS, based on the [BZ #nnn] convention in ChangeLog entries.  We should have it again for git.


Jim Meyering is the general expert in sourceware git setup, and may have
suggestions.  He is responsible for the current post-receive hook that
sends email, but that appears to be just a copy of a stock contrib script
distributed with git.

I looked at all the other projects' hooks/post-receive files and there
doesn't appear to be any that did anything interesting other than use the
same email-sending script we have now.

The old CVS setup was controlled by these lines in CVSROOT/loginfo:

# -T foo        distinguisher for tmp files
# -G glibc      bugzilla "product" field
# -m user@dom   email recipient
# -D domain     email domain sent from
# -U url        cvsweb url
# -C glibc      cvsweb ?cvsroot=<argument>
# -s %{sVv}     magic args
DEFAULT (/usr/sourceware/bin/log_accum_bugzillafied -T glibc -G glibc -C glibc -D sourceware.org -U "http://sourceware.org/cgi-bin/cvsweb.cgi/"; -m glibc-cvs@sourceware.org -s %{sVv})

That script can be seen at:
	http://sourceware.org/cgi-bin/cvsweb.cgi/infra/bin/log_accum_bugzillafied?rev=1.3&content-type=text/x-cvsweb-markup&cvsroot=sourceware

Ian Taylor was the last person to touch it, but I don't know who is the
best person to ask about any details.  It appears to work by a combination
of looking at the local database backing bugzilla, and sending formulaic
email to <sourceware-bugzilla@sourceware.org> to make the posts.

I don't really even know any more who to talk to about the bugzilla setup
itself.  So I guess the best place for someone to start here would be to
ping both Jim and overseers@sourceware.org about where to go next.

For stock solutions, a quick search found http://www.theoldmonk.net/gitzilla/
but I have no idea whether that's compatible with the sourceware bugzilla
installation or if the maintainers of sourceware bugzilla would want it
installed, or if it can be easily configured to work with the conventions
we use.

https://wiki.mozilla.org/Bugzilla:Addons#Integration_with_Source_Code_Management_utilities
points to a different thing, but via a link to a blog entry on a server
that says (in French) that the server is dead.
Comment 1 H.J. Lu 2012-06-21 17:16:24 UTC
http://www.theoldmonk.net/gitzilla/

says

Requirements

To install and run GitZilla, you need:

        Python (tested with 2.6.4, should work with >=2.5)
        pybugz (tested with 0.8.0)

Of course, to make it useful you also need a Bugzilla installation somewhere (not required to be on the same machine). GitZilla has been tested with Bugzilla 3.0.11 and should work with any Bugzilla version supported by pybugz.

The excellent pybugz can be obtained from http://github.com/ColdWind/pybugz/ and http://github.com/ColdWind/pybugz/downloads/
Comment 2 Joseph Myers 2013-03-20 12:25:01 UTC
Carlos, after the sourceware upgrade this should be unblocked (sourceware now has new-enough Python).
Comment 3 Carlos O'Donell 2013-03-20 14:51:20 UTC
I'm talking to overseers about enabling gitzilla integration so we can get commits into BZs.
Comment 4 Carlos O'Donell 2013-03-21 22:57:42 UTC
I hasn't been ruled out, but I have to see what we can do to convince overseers to let glibc prototype the use of gitzilla. It would be useful to all other sourceware projects using git. This isn't going to be solved shortly.
Comment 5 Tom Tromey 2013-08-12 09:04:36 UTC
It looked simple to me to hack up log_accum_bugzillafied
and the post-receive script.  I'm testing that combo now
as part of bug #14768.  There's no reason it wouldn't work
for glibc as well.
Comment 6 Tom Tromey 2013-10-15 13:38:15 UTC
(In reply to Tom Tromey from comment #5)
> It looked simple to me to hack up log_accum_bugzillafied
> and the post-receive script.  I'm testing that combo now
> as part of bug #14768.  There's no reason it wouldn't work
> for glibc as well.

I think this works pretty well now.
To enable it, on sourceware:

* Replace glibc.git/hooks/post-receive with a symlink to
  /usr/share/doc/git-core/contrib/hooks/post-receive-email
  This is derived from the same script currently used by glibc.

* Add some configury to glibc.git/config following the instructions
  in the new post-receive.  For glibc I think it is enough to add
  this to the [hooks] section:

   bugzillaproduct = glibc
Comment 7 Carlos O'Donell 2013-10-15 13:50:05 UTC
(In reply to Tom Tromey from comment #6)
> (In reply to Tom Tromey from comment #5)
> > It looked simple to me to hack up log_accum_bugzillafied
> > and the post-receive script.  I'm testing that combo now
> > as part of bug #14768.  There's no reason it wouldn't work
> > for glibc as well.
> 
> I think this works pretty well now.
> To enable it, on sourceware:
> 
> * Replace glibc.git/hooks/post-receive with a symlink to
>   /usr/share/doc/git-core/contrib/hooks/post-receive-email
>   This is derived from the same script currently used by glibc.
> 
> * Add some configury to glibc.git/config following the instructions
>   in the new post-receive.  For glibc I think it is enough to add
>   this to the [hooks] section:
> 
>    bugzillaproduct = glibc

Do you know this because you just enabled it for gdb and it works?

If it works for gdb I'd love to just turn it on for glibc.

Do I just ask overseers for help?
Comment 8 Tom Tromey 2013-10-15 14:00:16 UTC
(In reply to Carlos O'Donell from comment #7)

> 
> Do you know this because you just enabled it for gdb and it works?

I wrote it for the gdb/binutils git conversion and spent last week
trying it out.
It's possible that it still has bugs of course, but I think it is
reasonably well-debugged.

> If it works for gdb I'd love to just turn it on for glibc.
> 
> Do I just ask overseers for help?

I can do it if you like.
Comment 9 Carlos O'Donell 2013-10-15 15:01:34 UTC
(In reply to Tom Tromey from comment #8)
> (In reply to Carlos O'Donell from comment #7)
> 
> > 
> > Do you know this because you just enabled it for gdb and it works?
> 
> I wrote it for the gdb/binutils git conversion and spent last week
> trying it out.
> It's possible that it still has bugs of course, but I think it is
> reasonably well-debugged.
> 
> > If it works for gdb I'd love to just turn it on for glibc.
> > 
> > Do I just ask overseers for help?
> 
> I can do it if you like.

Please do. Between Roland, H.J. Lu, Joseph, and myself that's enough consensus to enable this for glibc. We're happy to work out any kinks.

Thanks!
Comment 10 jsm-csl@polyomino.org.uk 2013-10-15 15:27:50 UTC
On Tue, 15 Oct 2013, tromey at redhat dot com wrote:

> * Replace glibc.git/hooks/post-receive with a symlink to
>   /usr/share/doc/git-core/contrib/hooks/post-receive-email
>   This is derived from the same script currently used by glibc.

I don't see such a file on sourceware.  There are lots of 
post-receive-email files - which do you mean?

For glibc-cvs mails we show the complete diffs of the commit, but I don't 
think we want the full diffs (which can be very large) to show in Bugzilla 
comments.  Can it be configured to show less information in Bugzilla than 
in glibc-cvs?

How does this hook identify relevant bugs?  In ChangeLog entries in glibc 
we use [BZ #N]; conventions in git logs are less defined, but "bug N", 
"bug #N" and "BZ #N" are common.  We would in any case need to inform 
libc-alpha of the patterns to use in git logs, and update the contribution 
/ commit checklists accordingly.
Comment 11 Tom Tromey 2013-10-15 15:35:17 UTC
(In reply to joseph@codesourcery.com from comment #10)
> On Tue, 15 Oct 2013, tromey at redhat dot com wrote:
> 
> > * Replace glibc.git/hooks/post-receive with a symlink to
> >   /usr/share/doc/git-core/contrib/hooks/post-receive-email
> >   This is derived from the same script currently used by glibc.
> 
> I don't see such a file on sourceware.  There are lots of 
> post-receive-email files - which do you mean?

Sorry, I copied the wrong file name.
It is really /sourceware/infra/bin/post-receive-email

> For glibc-cvs mails we show the complete diffs of the commit, but I don't 
> think we want the full diffs (which can be very large) to show in Bugzilla 
> comments.  Can it be configured to show less information in Bugzilla than 
> in glibc-cvs?

Yes, from the comments in the file:

# hooks.bugzillashowrev
#   Like hooks.showrev, but only used for the bugzilla notification.
#   Defaults to hooks.showrev.

You'll want to test this, though, since I don't think I ended up using
this feature for src.git.

> How does this hook identify relevant bugs?  In ChangeLog entries in glibc 
> we use [BZ #N]; conventions in git logs are less defined, but "bug N", 
> "bug #N" and "BZ #N" are common.  We would in any case need to inform 
> libc-alpha of the patterns to use in git logs, and update the contribution 
> / commit checklists accordingly.

It does exactly what log_accum_bugzilla used to do, which is:

while ($log_txt =~ m/[^Aa](?:bug|PR|BZ)\s+\#?\s*(?:[a-z+-]+\/)?(?:\/)?(\d+)(.*)$/si) {
  $bug_id = $1;
  $log_txt = $2;

Somewhat unreadable I suppose.  I didn't edit the regexp when I
pulled out the bugzilla bits into a new script.
Comment 12 Tom Tromey 2013-10-18 15:56:23 UTC
I installed this with the settings:

	bugzillashowrev = "t=%s; printf 'https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=%%s' $t; echo; echo; git rev-list -1 --pretty $t"
	bugzillaproduct = glibc

This means the full patch should not show up in bugzilla.

Please give it a try and let me know if something is broken.
Comment 13 Carlos O'Donell 2013-10-18 16:27:24 UTC
(In reply to Tom Tromey from comment #12)
> I installed this with the settings:
> 
> 	bugzillashowrev = "t=%s; printf
> 'https://sourceware.org/git/gitweb.cgi?p=glibc.git;h=%%s' $t; echo; echo;
> git rev-list -1 --pretty $t"
> 	bugzillaproduct = glibc
> 
> This means the full patch should not show up in bugzilla.
> 
> Please give it a try and let me know if something is broken.

Tom,

Thank you very much for your work in getting this setup!

Cheers,
Carlos.
Comment 14 Andreas Schwab 2013-10-20 08:44:59 UTC
The mails are not properly encoded, see <http://sourceware.org/ml/glibc-bugs/2013-10/msg00238.html>.
Comment 15 Andreas Schwab 2013-10-24 08:21:42 UTC
Annotations should not be added for personal branches.

<http://sourceware.org/ml/glibc-bugs/2013-10/msg00298.html>
Comment 16 jsm-csl@polyomino.org.uk 2013-10-24 15:33:35 UTC
That is, they should be added for commits to master or release/*, but not 
any other branch (I think personal branches and company / distribution 
ones should be considered the same).  But all branches should continue to 
get the mails to glibc-cvs.
Comment 17 Carlos O'Donell 2013-10-24 16:46:00 UTC
I'm reopening to discuss the various tweaks we want to the auto-annotate.

* Wrong encoding
  - This is a valid issues that needs fixing.
  - Tom Tromey is aware of this and looking into it.

* Script handles push not commits
  - This results in very very large comments if the push is large.
  - See https://sourceware.org/bugzilla/show_bug.cgi?id=15698#c1
  - Script should split push into commits and post only the commit that
    made the change.
  - Carlos O'Donell is working on this.

* Script posts comments based on private branches.
  - This needs to go to libc-alpha for policy discussion.
  - Joseph could you please take this to libc-alpha to get consensus?
Comment 18 Tom Tromey 2013-10-24 19:30:39 UTC
(In reply to Andreas Schwab from comment #14)
> The mails are not properly encoded, see
> <http://sourceware.org/ml/glibc-bugs/2013-10/msg00238.html>.

Not really sure where the error creeps in.
It doesn't seem to be too likely to be the post-receive script.
Perhaps we need to emit a content-type header to say that
the text is utf-8.
Comment 19 jsm-csl@polyomino.org.uk 2013-10-25 15:15:47 UTC
On Thu, 24 Oct 2013, carlos at redhat dot com wrote:

> * Script posts comments based on private branches.
>   - This needs to go to libc-alpha for policy discussion.
>   - Joseph could you please take this to libc-alpha to get consensus?

My comment was simply a refinement of Andreas's.

I hope my more specific point that the comments need to identify the 
branch affected, which they don't at present, is obvious.
Comment 20 Tom Tromey 2013-10-25 21:05:04 UTC
(In reply to Tom Tromey from comment #18)
> (In reply to Andreas Schwab from comment #14)
> > The mails are not properly encoded, see
> > <http://sourceware.org/ml/glibc-bugs/2013-10/msg00238.html>.
> 
> Not really sure where the error creeps in.
> It doesn't seem to be too likely to be the post-receive script.
> Perhaps we need to emit a content-type header to say that
> the text is utf-8.

I believe I've fixed this, see
https://sourceware.org/bugzilla/show_bug.cgi?id=15670
Comment 21 Jackie Rosen 2014-02-16 18:25:21 UTC Comment hidden (spam)