[PATCH 2/2] gold: add a task in incremental build that can be used to check which inputs were modified

Mikolaj Zalewski mikolajz@google.com
Tue Mar 17 23:09:00 GMT 2009


  This patch adds a task that can be later used in an incremental
build to check if a full build is needed and what files needs to be
included. In an incremental build, it will be executed after all
Read_symbols and before Add_symbols, Add_archive_symbols and
Read_script. In non-incremental build, the order shouldn't change.
  There are two issues. Doing it after Read_symbols is suboptimal, as
it already loaded symbols from object files, including the ones that
were not modified. Later, I can divide this task into two and plug the
input checker between detecting the type of file and loading symbols.
A bigger problem is that we do it before Read_scripts thus we won't
see input files included by scripts. This can be fixed by dividing the
a Read_script task into one that reads script and another the applies
it to the layout. Before doing this, as a hack, I could treat every
file included by a script as modified.
  I don't know how plug-ins interacts with this - when do plugins add
input objects? Currently I print an error if they do it too late.
  BTW, `make check` in gold fails (in ver_test_4 assertion failed in
parameters.h:84 - gold_assert(this->options_valid())), but it also
fails without my patches - it's probably caused by an earlier patch. I
could try to look into it tomorrow.

2009-03-17  Mikolaj Zalewski  <mikolajz@google.com>
	* gold.cc (queue_initial_tasks): add Incremental_input_checker task.
	* incremental.cc (Incremental_input::report_archive): new method.
	(Incremental_inputs::report_object): new method.
	(Incremental_inputs::report_script): new method.
	(Incremental_inputs_checker::run): new method.
	* incremental.h: new file.
	* layout.cc (Layout::Layout): handle new incremental_inputs_.
	* layout.h (Layout::incremental_inputs): new method.
	(Layout::incremental_inputs_): new field.
	* plugin.cc (Plugin_manager::add_input_file): check incremental build state.
	* readsyms.cc (Read_symbols::locks): lock the Incremental_build_checker if any.
	(Read_symbols::do_read_symbols): use layout->incremental_inputs().
	* script.cc (read_input_scipt): add comment.
-------------- next part --------------
From a00b19a17dc4d8d849c91d1658d272820a0d3032 Mon Sep 17 00:00:00 2001
From: Mikolaj Zalewski <mikolajz@puchatek.dom>
Date: Tue, 17 Mar 2009 22:43:00 +0100
Subject: [PATCH] gold: add a task in incremental build that can be used to check which inputs were modifed

---
 gold/gold.cc        |   29 ++++++++++++-
 gold/incremental.cc |   31 +++++++++++++
 gold/incremental.h  |  118 +++++++++++++++++++++++++++++++++++++++++++++++++++
 gold/layout.cc      |    8 +++-
 gold/layout.h       |   10 ++++
 gold/plugin.cc      |   13 ++++++
 gold/readsyms.cc    |   26 +++++++++++-
 gold/script.cc      |    3 +
 8 files changed, 235 insertions(+), 3 deletions(-)

diff --git a/gold/gold.cc b/gold/gold.cc
index 93d0358..e150b1b 100644
--- a/gold/gold.cc
+++ b/gold/gold.cc
@@ -41,6 +41,7 @@
 #include "reloc.h"
 #include "defstd.h"
 #include "plugin.h"
+#include "incremental.h"
 
 namespace gold
 {
@@ -171,23 +172,49 @@ queue_initial_tasks(const General_options& options,
     thread_count = cmdline.number_of_input_files();
   workqueue->set_thread_count(thread_count);
 
-  // Read the input files.  We have to add the symbols to the symbol
+  // Read the input files.  This is done first by Read_symbols that checks
+  // if an input file is an archive, object file or script and queues an
+  // Add_archive_symbol, Add_symbols or Read_scipt task.  The latter tasks has
+  // to be executed in order, as we have to add the symbols to the symbol
   // table in order.  We do this by creating a separate blocker for
   // each input file.  We associate the blocker with the following
   // input file, to give us a convenient place to delete it.
   Task_token* this_blocker = NULL;
+
+  Incremental_inputs_checker *incremental_checker = NULL;
+  if (cmdline.options().incremental())
+    {
+      // In incremental link mode, the input checker to be executed before any
+      // task adding symbols.
+      this_blocker = new Task_token(true);
+      this_blocker->add_blocker();
+      incremental_checker = new Incremental_inputs_checker(this_blocker,
+        layout->incremental_inputs());
+      layout->incremental_inputs()->set_checker_task(incremental_checker);
+    }
+
   for (Command_line::const_iterator p = cmdline.begin();
        p != cmdline.end();
        ++p)
     {
       Task_token* next_blocker = new Task_token(true);
       next_blocker->add_blocker();
+      if (incremental_checker)
+        {
+          // The incremental input checker has to be executed after any
+          // Add_symbols or similar - add a blocker. Note that Read_symbols
+          // is not bolcked by incremental_checker.
+          incremental_checker->token()->add_blocker();
+        }
       workqueue->queue(new Read_symbols(input_objects, symtab, layout,
 					&search_path, 0, mapfile, &*p, NULL,
 					this_blocker, next_blocker));
       this_blocker = next_blocker;
     }
 
+  if (incremental_checker)
+    workqueue->queue(incremental_checker);
+
   if (options.has_plugins())
     {
       Task_token* next_blocker = new Task_token(true);
diff --git a/gold/incremental.cc b/gold/incremental.cc
index bd26bcd..12bd63c 100644
--- a/gold/incremental.cc
+++ b/gold/incremental.cc
@@ -22,6 +22,9 @@
 
 #include "gold.h"
 #include "elfcpp.h"
+#include "archive.h"
+#include "incremental.h"
+#include "object.h"
 
 using elfcpp::Convert;
 
@@ -142,5 +145,33 @@ class Incremental_input_entry_write
   internal::Incremental_input_entry_data* p_;
 };
 
+void
+Incremental_inputs::report_archive(const Input_argument*, Archive* archive)
+{
+  // TODO: implement
+  printf("TODO: make use of: %s is archive\n", archive->filename().c_str());
+}
+
+void
+Incremental_inputs::report_object(const Input_argument*, Object* object)
+{
+  // TODO: implement
+  printf("TODO: make use of: %s is object\n", object->name().c_str());
+}
 
+void
+Incremental_inputs::report_script(const Input_argument*, void*)
+{
+  // TODO: implement
+  printf("TODO: handle script arguments\n");
 }
+
+void
+Incremental_inputs_checker::run(Workqueue*)
+{
+  // TODO: implement
+  printf("TODO: Incremental check task\n");
+  this->inputs_->set_checker_task(NULL);
+}
+
+}  // End namespace gold.
diff --git a/gold/incremental.h b/gold/incremental.h
new file mode 100644
index 0000000..fc3503c
--- /dev/null
+++ b/gold/incremental.h
@@ -0,0 +1,118 @@
+// inremental.cc -- incremental linking support for gold
+
+// Copyright 2009 Free Software Foundation, Inc.
+// Written by Mikolaj Zalewski <mikolajz@google.com>.
+
+// This file is part of gold.
+
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 3 of the License, or
+// (at your option) any later version.
+
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with this program; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston,
+// MA 02110-1301, USA.
+
+#ifndef GOLD_INCREMENTAL_H
+#define GOLD_INCREMENTAL_H
+
+#include <vector>
+#include "workqueue.h"
+
+namespace gold
+{
+
+class Archive;
+class Input_argument;
+class Incremental_inputs_checker;
+class Object;
+
+// This class contains the information needed during an incremental
+// build about the inputs necessary to build the .gnu_incremental_inputs.
+class Incremental_inputs
+{
+ public:
+  Incremental_inputs() { }
+  ~Incremental_inputs() { }
+
+  // Called by Read_symbols to report that ARGUMENT is an achive
+  // ARCHIVE.
+  void
+  report_archive(const Input_argument* argument, Archive* archive);
+
+  // Called by Read_symbols to report that ARGUMENT is an object
+  // file OBJECT.
+  void
+  report_object(const Input_argument* argument, Object* object);
+
+  // Called by Read_symbols to report that ARGUMENT is a script
+  // (details to be done).
+  void
+  report_script(const Input_argument* argument, void* todo);
+
+  // Pointer to the checker task if it is not yet run or NULL otherwise.
+  Incremental_inputs_checker*
+  checker_task()
+  { return this->checker_; }
+
+  void
+  set_checker_task(Incremental_inputs_checker* checker)
+  { this->checker_ = checker; }
+
+ private:
+  Incremental_inputs_checker* checker_;
+};
+
+// A task checking that the .gnu_incremental_inputs section of the target file
+// (if exists), checking what needs to be rebuilt
+class Incremental_inputs_checker : public Task
+{
+ public:
+  Incremental_inputs_checker(Task_token* blocked_token,
+                             Incremental_inputs* inputs)
+    : blocked_token_(blocked_token), inputs_(inputs)
+  { this->token_ = new Task_token(true); }
+
+  ~Incremental_inputs_checker()
+  { delete this->token_; }
+
+  // The standard task methods.
+
+  Task_token*
+  is_runnable()
+  { if (this->token_->is_blocked()) return this->token_; return NULL;  }
+
+  void
+  locks(Task_locker* tl)
+  { tl->add(this, this->blocked_token_); }
+
+  void
+  run(Workqueue*);
+
+  // Token blocking this task.
+
+  Task_token*
+  token()
+  { return this->token_; }
+
+protected:
+  std::string
+  get_name() const
+  { return std::string("Incremental_input_checker"); }
+
+private:
+  Task_token* token_;
+  Task_token* blocked_token_;
+  Incremental_inputs* inputs_;
+};
+
+} // End namespace gold.
+
+#endif // !defined(GOLD_INCREMENTAL_H)
diff --git a/gold/layout.cc b/gold/layout.cc
index b59caea..3e3f7a4 100644
--- a/gold/layout.cc
+++ b/gold/layout.cc
@@ -48,6 +48,7 @@
 #include "descriptors.h"
 #include "layout.h"
 #include "plugin.h"
+#include "incremental.h"
 
 namespace gold
 {
@@ -121,7 +122,8 @@ Layout::Layout(const General_options& options, Script_options* script_options)
     input_with_gnu_stack_note_(false),
     input_without_gnu_stack_note_(false),
     has_static_tls_(false),
-    any_postprocessing_sections_(false)
+    any_postprocessing_sections_(false),
+    incremental_inputs_(NULL)
 {
   // Make space for more than enough segments for a typical file.
   // This is just for efficiency--it's OK if we wind up needing more.
@@ -130,6 +132,10 @@ Layout::Layout(const General_options& options, Script_options* script_options)
   // We expect two unattached Output_data objects: the file header and
   // the segment headers.
   this->special_output_list_.reserve(2);
+
+  // Initialize structure needed for an incremental build.
+  if (options.incremental())
+    this->incremental_inputs_ = new Incremental_inputs;
 }
 
 // Hash a key we use to look up an output section mapping.
diff --git a/gold/layout.h b/gold/layout.h
index 010ab2e..809255b 100644
--- a/gold/layout.h
+++ b/gold/layout.h
@@ -39,6 +39,7 @@ namespace gold
 {
 
 class General_options;
+class Incremental_inputs;
 class Input_objects;
 class Mapfile;
 class Symbol_table;
@@ -368,6 +369,12 @@ class Layout
   script_options() const
   { return this->script_options_; }
 
+  // Return the object managing inputs in incremental build. NULL in
+  // non-incremental builds.
+  Incremental_inputs*
+  incremental_inputs()
+  { return this->incremental_inputs_; }
+
   // Compute and write out the build ID if needed.
   void
   write_build_id(Output_file*) const;
@@ -715,6 +722,9 @@ class Layout
   bool has_static_tls_;
   // Whether any sections require postprocessing.
   bool any_postprocessing_sections_;
+  // In incremental build, holds information check the inputs and build the
+  // .gnu_incremental_inputs section.
+  Incremental_inputs* incremental_inputs_;
 };
 
 // This task handles writing out data in output sections which is not
diff --git a/gold/plugin.cc b/gold/plugin.cc
index 3c4d4ae..66054a6 100644
--- a/gold/plugin.cc
+++ b/gold/plugin.cc
@@ -38,6 +38,7 @@
 #include "readsyms.h"
 #include "symtab.h"
 #include "elfcpp.h"
+#include "incremental.h"
 
 namespace gold
 {
@@ -407,6 +408,18 @@ Plugin_manager::add_input_file(char *pathname)
   Input_argument* input_argument = new Input_argument(file);
   Task_token* next_blocker = new Task_token(true);
   next_blocker->add_blocker();
+  if (this->layout_->incremental_inputs())
+    {
+      // FIXME: when do plugins add input files? Can be just print an error if
+      // they do it too late, or should we somehow handle it?
+      Incremental_inputs_checker* checker;
+      checker = this->layout_->incremental_inputs()->checker_task(); 
+      if (checker)
+        checker->token()->add_blocker();
+      else
+        gold_error(_("Input added by plug-in too late for incremental build. "
+            "Don't use --incremental with this plug-in."));
+    }
   this->workqueue_->queue_soon(new Read_symbols(this->input_objects_,
                                                 this->symtab_,
                                                 this->layout_,
diff --git a/gold/readsyms.cc b/gold/readsyms.cc
index dc85898..8aa753f 100644
--- a/gold/readsyms.cc
+++ b/gold/readsyms.cc
@@ -33,6 +33,8 @@
 #include "script.h"
 #include "readsyms.h"
 #include "plugin.h"
+#include "layout.h"
+#include "incremental.h"
 
 namespace gold
 {
@@ -147,8 +149,14 @@ Read_symbols::is_runnable()
 // locks here.
 
 void
-Read_symbols::locks(Task_locker*)
+Read_symbols::locks(Task_locker* tl)
 {
+  // FIXME: Read_symbols for inputs added by scripts will be called
+  // too late, after Incremental_inputs_checker::run has been called.
+  // We need to check if the token is still present.
+  if (this->layout_->incremental_inputs() &&
+      this->layout_->incremental_inputs()->checker_task())
+    tl->add(this, this->layout_->incremental_inputs()->checker_task()->token());
 }
 
 // Run a Read_symbols task.
@@ -169,6 +177,8 @@ Read_symbols::run(Workqueue* workqueue)
 bool
 Read_symbols::do_read_symbols(Workqueue* workqueue)
 {
+  Incremental_inputs* incremental_inputs = this->layout_->incremental_inputs();
+
   if (this->input_argument_->is_group())
     {
       gold_assert(this->input_group_ == NULL);
@@ -214,6 +224,9 @@ Read_symbols::do_read_symbols(Workqueue* workqueue)
 	  // Unlock the archive so it can be used in the next task.
 	  arch->unlock(this);
 
+	  if (incremental_inputs)
+	    incremental_inputs->report_archive(this->input_argument_, arch);
+
 	  workqueue->queue_next(new Add_archive_symbols(this->symtab_,
 							this->layout_,
 							this->input_objects_,
@@ -241,6 +254,9 @@ Read_symbols::do_read_symbols(Workqueue* workqueue)
           // We are done with the file at this point, so unlock it.
           obj->unlock(this);
 
+          if (incremental_inputs)
+            incremental_inputs->report_object(this->input_argument_, obj);
+
           workqueue->queue_next(new Add_symbols(this->input_objects_,
                                                 this->symtab_,
                                                 this->layout_,
@@ -287,6 +303,9 @@ Read_symbols::do_read_symbols(Workqueue* workqueue)
 	      return false;
 	    }
 
+	  if (incremental_inputs)
+            incremental_inputs->report_object(this->input_argument_, obj);
+
 	  Read_symbols_data* sd = new Read_symbols_data;
 	  obj->read_symbols(sd);
 
@@ -325,6 +344,11 @@ Read_symbols::do_read_symbols(Workqueue* workqueue)
   // read multiple scripts simultaneously, which could lead to
   // unpredictable changes to the General_options structure.
 
+  if (incremental_inputs)
+    {
+      // TODO: handle scripts.
+      incremental_inputs->report_script(this->input_argument_, NULL);
+    }
   workqueue->queue_soon(new Read_script(this->symtab_,
 					this->layout_,
 					this->dirpath_,
diff --git a/gold/script.cc b/gold/script.cc
index 44e43f7..451c18b 100644
--- a/gold/script.cc
+++ b/gold/script.cc
@@ -1408,6 +1408,9 @@ read_input_script(Workqueue* workqueue, Symbol_table* symtab, Layout* layout,
 	  nb = new Task_token(true);
 	  nb->add_blocker();
 	}
+      // FIXME: in incremental build, this happens after
+      // Incremental_inputs_checker run - no incremental build data
+      // for such files.
       workqueue->queue_soon(new Read_symbols(input_objects, symtab,
 					     layout, dirsearch, 0, mapfile, &*p,
 					     input_group, this_blocker, nb));
-- 
1.4.4.2



More information about the Binutils mailing list