RFC: The Binutils and LLM generated patches

[Nick Clifton]

Hi Mark,

>> * Using LLMs to assist in writing code is fine providing that the LLM does not actually generate code.  So for example using an LLM for translations is OK.  So is
>> using an LLM to provide text to speech services or to search for information, and so on.
> 
> I think I understand what you are trying to say here. If I understand you correctly you want to explain that using an LLM as a tool that doesn't generate "content" is
> fine. As if you read an article on a certain algorithm or use it as a kind of search engine.

Correct.

> But I am a little confused about the "for translations".

I was actually thinking of using LLMs to translate articles, web pages
documents etc into a language with which the submitter was familiar.
But I agree that my wording was unclear.


 > I assume you
> aren't suggesting the project would accept LLM generated patches 
 > for translations of binutils? Should we coordinate this with the
 > translation project teams?

I think that we should continue to use the Translation Project for
translations of strings/text in the GNU Binutils project.   And only
use that source for our translations.  But I am also open to the
idea that the Translation Project *might* decide to accept machine
generated translations.  If they do, then that is fine with me, but
I am not going pressure them either way.

How about just removing a mention of translations from the paragraph.
ie:

   * Using LLMs to assist in writing code is fine providing that the
     LLM does not actually generate code.  So for example using an
     LLM to provide text to speech services or to search for published
     information is OK.


>> When submitting a non-legally-significant LLM generated change, it is still necessary to clearly indicate the use of the LLM.
> 
> But I don't think this is necessary or even desirable.

I like it because I think that we need to set the precedent that machine
generated content must be identified as such, even if it is for something
that is not legally significant.  I do foresee a future where some LLM
generated content becomes acceptable and I would like to have labelling
such content as an already accepted practice.

> And I am afraid people will use it for advertising the
> latest hip model/company.

I am not worried about that.  If people want to contribute to the project
as a means to advertise the latest hip model/company then let them.  As
long as the submissions contribute to the overall improvement of the
project then I am not worried about the motives for those contributions.


>> Generated-By: GNU-LLM version 1.0 LLM-Query: Create a program to solve all GNU problems
> 
> While something like this might be useful when using some deterministic tools I think it isn't useful for LLMs.

Yeah, Sid also mentioned this.  I have now added this paragraph to my
local copy of the text:

   The LLM-Query needs to be recorded for auditing purposes,
   but is not expected to produce the exact same output due
   to the non-deterministic nature of LLMs.


>> In addition all patch submissions must involve a human.  Fully automated patch submission, whether by a bot, a script, or some other means is not acceptable.
> 
> Maybe extend this to bug reports?

Hmm, I think not.  The reason being that I do not think that bug
reports, per se, need to be covered by copyright.  Bug reports
that include proposed fixes would need to abide by these rules,
but a machine generated bug report - without a legally significant
proposed fix - are OK.


>> The copyright assignment or DCO allows the GNU Binutils project to trust that the submitter is the author of the submission, and that they have the right under
>> copyright law to make the submission.
> 
> Agreed, but I maybe reformulate the second paragraph to make clear what this means in practice (for this project). How about:
> 
> So for each submission someone has to certify users gets all the rights granted by the GNU Binutils project license, the GNU General Public License (see COPYING3).

I think that we do need to explicitly mention that the submitter
is legally able to make the contribution.  But I also agree that
we should make clear that they are granting the use of this
copyrighted material to the project.  So how about this:

   The copyright assignment or DCO allows the GNU Binutils
   project to trust that the submitter is legally able to
   make the contribution and that the submission can be used
   under the terms of the GNU General Public License (see the
   COPYING3 file).


>> The policy uses the term "LLM generated" rather than "A.I. generated" as the later could be misunderstood.  See [2] for more details.
> 
> I agree with this. But find the reference a little confusing and negative. I would drop it or try to find a more positive definition of LLM.

Eh, I am ambivalent on this issue.  I included it because the
FSF obviously have a stance on the wording, and I want to keep
them happy.  My thoughts were that if readers object to the
characterisations of the term "A.I." in the link then they can
always talk to the FSF directly.

Cheers
   Nick