[PATCH v5 0/10 Add support for CRC64 generation in linker

[Ulf Samuelsson]

Discovered two bugs (which I fixed locally) so any review should be for 
general concepts.

I will modify the tests so that the CRCs will be computed
on known test data.

As an example:
The CRC64-ECMA run over the 9 characters in the string "123456789"
should generate 0x6C40DF5F0B497347.

So the revised test for CRC64-ECMA will calculate over this string
and store the result, which means that most of the test data
can be ignored.

Here I store the CRC at 0x1120, and the expected value in 0x1110.

      5     1100 434f4445 deadbeef 00000000 00000000 CODE............
      6     1110 4773490b 5fdf406c 00000000 00000000 GsI._.@l........
      7     1120 4773490b 5fdf406c 00000000 00000000 GsI._.@l........
      8     1130 31323334 35363738 3900ffff ffffffff 123456789.......

They match, so this seems to work.

Best Regards

Ulf Samuelsson


On 2023-02-22 17:15, Ulf Samuelsson via Binutils wrote:
> Fifth patchset.
>
> +* The linker script has a new command to insert a timestamp
> +  TIMESTAMP
> +  inserts the current time (seconds since Epoch) as a 64-bit value
> +
> +* The linker script syntax has new commands for debugging a linker script
> +  DEBUG ON  turns on debugging
> +  DEBUG OFF turns off debugging
> +
> +* The linker script syntax has new commands for handling CRC-32/64 calculations
> +  on the '.text' section
> +  It uses code from https://www.libcrc.org/
> +
> +  DIGEST "CRC32"              (start, end)
> +  DIGEST "CRC64-ECMA"         (start, end)
> +  DIGEST "CRC64-ISO"          (start, end)
> +  DIGEST "CRC64-WE"           (start, end)
> +  DIGEST POLY (size, polynome)(start, end)
> +  DIGEST POLYI(size, polynome)(start, end) (inversion during CRC calculation)
> +  DIGEST TABLE
> +  DIGEST SECTION "<section>
>
> ran indent on the new files.
>
> The *.d files in the testsuite was edited to
> reduce size and to remove all '(' characters.
> The new linker can link using the testsuite/ld-script/*.t
> linker files, but they do not pass the 'make check-ld'
> test.
>
> Fourth patchset.
> Get rid of binaries in testsuite.
>
> Here is the third patchset for introducing CRC64 generation.
> This has focused on supporting maintainability of the feature.
> In order to do so, CRC32 generation has been introduced to
> detect issues when extending the linker with another algoritm
> for checking integrity of the image.
> This will simplify adding other algorithms like SHA algoritms
> in the future.
>
> In addition, the TIMESTAMP command is used to store the current time
> in seconds since Epoch is stored in the image.
>
> The main routine for calculating CRCs has been broken down
> in subroutines which allow other algorithms.
>
> An algorithm typically needs
> <type> *init_<algo>_tab( <type> poly )
> <type> calc_<algo>_inv( const unsigned char *input_str, size_t num_bytes );
> <type> calc_<algo>    ( const unsigned char *input_str, size_t num_bytes );
> void lang_add_<algo>_syndrome(bool invert, bfd_vma poly)
> void lang_add_<algo>_table(void)
>
> The common functions are:
> * bool get_text_section_contents(void)
>    This reads the '.text' section and its contents
>    Pointers are stored in the global variables
>    asection *text_section
>    char     *text_contents
> * bool set_crc_checksum(bfd_vma crc, bfd_vma addr, bfd_vma size)
>    Stores the CRC at the index addr. The size of the CRC in bytes is specified.
>    Storing something larger that 64-bit is not supported here.
> * bool symbol_lookup(char *name, bfd_vma *val)
>    Gets the value of a symbol into 'val'. Returns false if not found.
>
> To add CRC32 support means adding the following commands
> * CRC32 CRC32                    '(' crc_start ',' crc_end ')'
> * CRC32 POLY  '[' mustbe_exp ']' '(' crc_start ',' crc_end ')'
> * CRC32 POLYI '[' mustbe_exp ']' '(' crc_start ',' crc_end ')'
>
> ================
> Here is the second patchset for introducing CRC64 generation in the linker
> This is mainly looking at indentation and whitespace errors.
> The patches applies cleanly without whitespace error
> except for the last patch which contains the testsuite.
> When the contents of .text is printed out, sometimes
> the last byte of a line in the CRC table is a space.
> This is reported as a whitespace error.
>
> Supports the following new linker commands:
> * CRC64 ECMA                     '(' crc_start ',' crc_end ')'
> * CRC64 ISO                      '(' crc_start ',' crc_end ')'
> * CRC64 POLY  '[' mustbe_exp ']' '(' crc_start ',' crc_end ')'
> * CRC64 POLYI '[' mustbe_exp ']' '(' crc_start ',' crc_end ')'
> * CRC64 TABLE
>
> ECMA  == 0x42F0E1EBA9EA3693
> ISO   == 0xD800000000000000
> POLY  == Allows your own polynome
> POLYI == Allows your own polynome, with inversion during calc
>
> The CRC is calculated from crc_start to crc_end (not included)
>
> The "CRC64 <polynome> command
> * Allows specifying the polynom (ECMA(default), ISO or your own)
> * Allows for inversion in the CRC calculation (CRC64-WE)
> * Allows specifying the area that should be checked.
> * reserves room for the CRC (8 bytes)
> * Declares a symbol ___CRC64___ for the address of the CRC.
> * Declares a symbol ___CRC64_START___ for the first address of the checked area
> * Declares a symbol ___CRC64_END___ for the first address after the checked area
>
> The symbols names are declared in "checksum.h".
> If the names are unsuitable, it is easy to change.
>
> The CRC TABLE command
>    This is used to speed up the CRC calculation.
> * Creates a 2kB table which speeds up the CRC calculation
> * Can insert the 2kB table into the .text section at current location.
> * Declares a symbol ___CRC64_TABLE___ if the table is inserted.
>
> Comments on the code:
>
> The process of calculation involves:
> * Check if CRC is requested
> * Validation of CRC prerequisites
> * get .text section
> * get .text section contents
> * calculate CRC over the area
> * insert the CRC in the correct location
> ====================================
>
> This version also supports the
> * DEBUG ON
> * DEBUG OFF
> turning on/off yydebug
> I find it useful for debugging the build, but it can easly be removed.
>
> This patch contains a lot of debug output, that is enabled by
> #define DEBUG_CRC 1
> This should be removed in the final version.
>
> The ld.texi stuff needs some more work. Not very experienced with that.
>
> Added an entry in NEWS
>
> Added 4 test cases for the different CRC64 polynome commands.
> All testcases generate a CRC table.
>
> The code is using the libcrc released under an MIT license found in
> * https://www.libcrc.org/
> * https://github.com/lammertb/libcrc/tree/v2.0
> Author:  Lammert Bies
> A license for libcrc is added to the patch.
>
> [PATCH v5 01/10] TIMESTAMP: ldlang: process
> [PATCH v5 02/10] DIGEST: NEWS
> [PATCH v5 03/10] DIGEST: ld.texi
> [PATCH v5 04/10] LIBCRC: license
> [PATCH v5 05/10] DIGEST: ldlex.l
> [PATCH v5 06/10] DIGEST: ldgram.y
> [PATCH v5 07/10] DIGEST: Makefile.am: add new files
> [PATCH v5 08/10] DIGEST: CRC-32/64 algorithms
> [PATCH v5 09/10] DIGEST: ldmain.c: add CRC calculation
> [PATCH v5 10/10] DIGEST: testsuite
>