This is the mail archive of the libc-alpha@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

[PATCH] localedef: Add --no-warnings/--warnings option


>From localedef --help:

Output control:
...
      --no-warnings=<warnings>   Comma-separated list of warnings to disable;
                             supported warnings are: ascii, intcurrsym
...
      --warnings=<warnings>  Comma-separated list of warnings to enable;
                             supported warnings are: ascii, intcurrsym

Locales using SHIFT_JIS and SHIFT_JISX0213 character maps are not ASCII
compatible. In order to build locales using these character maps, and
have localedef exit with a status of 0, we add new option to localedef
to disable or enable specific warnings. The options are --no-warnings
and --warnings, to disable and enable specific warnings respectively.
The options take a comma-separated list of warning names. The warning
names are taken directly from the generated warning.  When a warning
that can be disabled is issued it will print something like this: foo is
not defined [--no-warnings=foo]

For the initial implementation we add two controllable warnings; first
'ascii' which is used by the localedata installation makefile target to
install SHIFT_JIS and SHIFT_JISX0213-using locales without error; second
'intcurrsym' which allows a program to use a non-standard international
currency symbol without triggering a warning.  The 'intcurrsym' is
useful in the future if country codes are added that are not in our
current ISO 4217 list, and the user wants to avoid the warning. Having
at least two warnings to control gives an example for how the changes
can be extended to more warnings if required in the future.

These changes allow ja_JP.SHIFT_JIS and ja_JP.SHIFT_JISX0213 to be
compiled without warnings using --no-warnings=ascii. The
localedata/Makefile $(INSTALL-SUPPORTED-LOCALES) target is adjusted to
automatically add `--no-warnings=ascii` for such charmaps, and likewise
localedata/gen-locale.sh is adjusted with similar logic.

Signed-off-by: Carlos O'Donell <carlos@redhat.com>
---
 ChangeLog                     | 25 +++++++++++++++++
 locale/programs/charmap.c     | 12 ++++++--
 locale/programs/charmap.h     |  2 ++
 locale/programs/ld-monetary.c |  9 ++++--
 locale/programs/localedef.c   | 64 +++++++++++++++++++++++++++++++++++++++++++
 locale/programs/localedef.h   |  1 +
 localedata/Makefile           | 12 ++++++--
 localedata/gen-locale.sh      | 24 ++++++++++------
 8 files changed, 134 insertions(+), 15 deletions(-)

-- 
Cheers,
Carlos.
>From 9c9fd183d1aa017ade39c60a9dfe9ed0f9759eb1 Mon Sep 17 00:00:00 2001
From: Carlos O'Donell <carlos@systemhalted.org>
Date: Tue, 17 Oct 2017 01:33:42 -0700
Subject: [PATCH] localedef: Add --no-warnings/--warnings option

>From localedef --help:

Output control:
...
      --no-warnings=<warnings>   Comma-separated list of warnings to disable;
                             supported warnings are: ascii, intcurrsym
...
      --warnings=<warnings>  Comma-separated list of warnings to enable;
                             supported warnings are: ascii, intcurrsym

Locales using SHIFT_JIS and SHIFT_JISX0213 character maps are not ASCII
compatible. In order to build locales using these character maps, and
have localedef exit with a status of 0, we add new option to localedef
to disable or enable specific warnings. The options are --no-warnings
and --warnings, to disable and enable specific warnings respectively.
The options take a comma-separated list of warning names. The warning
names are taken directly from the generated warning.  When a warning
that can be disabled is issued it will print something like this: foo is
not defined [--no-warnings=foo]

For the initial implemenation we add two controllable warnings; first
'ascii' which is used by the localedata installation makefile target to
install SHIFT_JIS and SHIFT_JISX0213-using locales without error; second
'intcurrsym' which allows a program to use a non-standard international
currency symbol without triggering a warning.  The 'intcurrsym' is
useful in the future if country codes are added that are not in our
current ISO 4217 list, and the user wants to avoid the warning. Having
at least two warnings to control gives an example for how the changes
can be extended to more warnings if required in the future.

These changes allow ja_JP.SHIFT_JIS and ja_JP.SHIFT_JISX0213 to be
compiled without warnings using --no-warnings=ascii. The
localedata/Makefile $(INSTALL-SUPPORTED-LOCALES) target is adjusted to
automatically add `--no-warnings=ascii` for such charmaps, and likewise
localedata/gen-locale.sh is adjusted with similar logic.

Signed-off-by: Carlos O'Donell <carlos@redhat.com>
---
 ChangeLog                     | 25 +++++++++++++++++
 locale/programs/charmap.c     | 12 ++++++--
 locale/programs/charmap.h     |  2 ++
 locale/programs/ld-monetary.c |  9 ++++--
 locale/programs/localedef.c   | 64 +++++++++++++++++++++++++++++++++++++++++++
 locale/programs/localedef.h   |  1 +
 localedata/Makefile           | 12 ++++++--
 localedata/gen-locale.sh      | 24 ++++++++++------
 8 files changed, 134 insertions(+), 15 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 2727111..b5c004b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,28 @@
+2017-10-16  Carlos O'Donell  <carlos@redhat.com>
+
+	* locale/programs/localedef.c (warn_ascii): Declare.
+	(warn_int_curr_symbol): Declare.
+	(OPT_NO_WARN): Define.
+	(OPT_WARN): Define.
+	(options): Add entry for --no-warnings, and --warnings.
+	(set_warnings): New function to enable/disable warnings.
+	(parse_opt): Call set_warnings for OPT_NO_WARN and OPT_WARN.
+	* locale/programs/localedef.h (warn_int_curr_symbol): Declare extern
+	bool.
+	* locale/programs/ld-monetary.c (monetary_finish): If
+	warn_int_curr_symbol is true then record a warning about the symbol
+	not being in our ISO 4217 list.
+	* locale/programs/charmap.c (charmap_read): If warn_ascii is true then
+	record a warning about ASCII compatibility.
+	* locale/programs/charmap.h: Add tenative definition of warn_ascii.
+	* localedata/gen-locale.sh: Default flags to `--quiet -c'.
+	Add `--no-warnings=ascii' to locales using SHIFT_JIS or SHIFT_JIXX0213.
+	Pass flags to generate_locale.
+	(generate_locale): Accept new flag argument and pass it to localedef
+	invocation.
+	* localedata/Makefile (INSTALL-SUPPORTED-LOCALES): Use
+	--no-warnings=ascii for SHIFT_JIS and SHIFT_JISX0213 charmaps.
+
 2017-10-15  Carlos O'Donell  <carlos@redhat.com>
 
 	* localedata/collate-test.c (allocate_arrays): Don't use \n in
diff --git a/locale/programs/charmap.c b/locale/programs/charmap.c
index a670db9..964d932 100644
--- a/locale/programs/charmap.c
+++ b/locale/programs/charmap.c
@@ -256,9 +256,15 @@ charmap_read (const char *filename, int verbose, int error_not_found,
 
       if (failed)
 	{
-	  record_warning (_("\
-character map `%s' is not ASCII compatible, locale not ISO C compliant\n"),
-			  result->code_set_name);
+	  /* A user may disable the ASCII compatibility warning check,
+	     but we must remember that the encoding is not ASCII
+	     compatible, since it may have other implications.  Later
+	     we will set _NL_CTYPE_MAP_TO_NONASCII from this value.  */
+	  if (warn_ascii)
+	    record_warning (_(
+"character map `%s' is not ASCII compatible, locale not ISO C compliant "
+"[--no-warnings=ascii]"),
+			    result->code_set_name);
 	  enc_not_ascii_compatible = true;
 	}
     }
diff --git a/locale/programs/charmap.h b/locale/programs/charmap.h
index 5d6b48f..441d429 100644
--- a/locale/programs/charmap.h
+++ b/locale/programs/charmap.h
@@ -66,6 +66,8 @@ struct charseq
 
 /* True if the encoding is not ASCII compatible.  */
 extern bool enc_not_ascii_compatible;
+/* True if the ASCII compatibility check should raise a warning.  */
+bool warn_ascii;
 
 
 /* Prototypes for charmap handling functions.  */
diff --git a/locale/programs/ld-monetary.c b/locale/programs/ld-monetary.c
index 9d94738..71df376 100644
--- a/locale/programs/ld-monetary.c
+++ b/locale/programs/ld-monetary.c
@@ -234,12 +234,17 @@ No definition for %s category found"), "LC_MONETARY");
 	  char symbol[4];
 	  strncpy (symbol, monetary->int_curr_symbol, 3);
 	  symbol[3] = '\0';
+	  /* A user may disable this waning for testing purposes or
+	     for building a locale with a 3 digit country code that
+	     was not yet supported in our ISO 4217 list.
+	     See the use of --no-warnings=intcurrsym.  */
 	  if (bsearch (symbol, valid_int_curr, NR_VALID_INT_CURR,
 		       sizeof (const char *),
-		       (comparison_fn_t) curr_strcmp) == NULL)
+		       (comparison_fn_t) curr_strcmp) == NULL
+	      && warn_int_curr_symbol)
 	    record_warning (_("\
 %s: value of field `int_curr_symbol' does \
-not correspond to a valid name in ISO 4217"),
+not correspond to a valid name in ISO 4217 [--no-warnings=intcurrsym]"),
 			    "LC_MONETARY");
 	}
     }
diff --git a/locale/programs/localedef.c b/locale/programs/localedef.c
index 7d76154..a5e474b 100644
--- a/locale/programs/localedef.c
+++ b/locale/programs/localedef.c
@@ -32,6 +32,7 @@
 #include <error.h>
 #include <sys/mman.h>
 #include <sys/stat.h>
+#include <ctype.h>
 
 #include "localedef.h"
 #include "charmap.h"
@@ -51,6 +52,13 @@ int posix_conformance;
 /* If not zero give a lot more messages.  */
 int verbose;
 
+/* Warnings which can be disabled:  */
+/* By default we check the character map for ASCII compatibility.  */
+bool warn_ascii = true;
+/* By default we check that the international currency symbol matches a
+   known country code.  */
+bool warn_int_curr_symbol = true;
+
 /* Warnings recorded by record_warnings (see localedef.h).  */
 int recorded_warning_count;
 
@@ -114,6 +122,8 @@ void (*argp_program_version_hook) (FILE *, struct argp_state *) = print_version;
 #define OPT_LIST_ARCHIVE 309
 #define OPT_LITTLE_ENDIAN 400
 #define OPT_BIG_ENDIAN 401
+#define OPT_NO_WARN 402
+#define OPT_WARN 403
 
 /* Definitions of arguments for argp functions.  */
 static const struct argp_option options[] =
@@ -134,6 +144,13 @@ static const struct argp_option options[] =
   { "quiet", OPT_QUIET, NULL, 0,
     N_("Suppress warnings and information messages") },
   { "verbose", 'v', NULL, 0, N_("Print more messages") },
+  { "no-warnings", OPT_NO_WARN, N_("<warnings>"), 0,
+    N_("Comma-separated list of warnings to disable; "
+       "supported warnings are: ascii, intcurrsym") },
+  { "warnings", OPT_WARN, N_("<warnings>"), 0,
+    N_("Comma-separated list of warnings to enable; "
+       "supported warnings are: ascii, intcurrsym") },
+
   { NULL, 0, NULL, 0, N_("Archive control:") },
   { "no-archive", OPT_NO_ARCHIVE, NULL, 0,
     N_("Don't add new data to archive") },
@@ -309,6 +326,45 @@ no output file produced because errors were issued"));
   exit (recorded_warning_count != 0);
 }
 
+/* Search warnings for matching warnings and if found enable those
+   warnings if ENABLED is true, otherwise disable the warnings.  */
+static void
+set_warnings (char *warnings, bool enabled)
+{
+  char *tok;
+  char *save;
+  char *copy = (char *) malloc (strlen (warnings) + 1);
+
+  /* Remove all spaces from the warnings list to make the processing
+     a more robust.  We don't support spaces in a warning name.  */
+
+  save = copy;
+  tok = warnings;
+
+  do {
+    while (isspace (*tok))
+      tok++;
+  } while ((*save++ = *tok++));
+
+  warnings = copy;
+
+  /* Tokenize the input list of warnings to set, compare them to
+     known warnings, and set the warning.  We purposely ignore unknown
+     warnings, and are thus forward compatible, users can attempt to
+     disable whaterver new warnings they know about, but we will only
+     disable those *we* known about.  */
+
+  while ((tok = strtok_r (warnings, ",", &save)) != NULL)
+    {
+      warnings = NULL;
+      if (strcmp (tok, "ascii") == 0)
+	warn_ascii = enabled;
+      else if (strcmp (tok, "intcurrsym") == 0)
+	warn_int_curr_symbol = enabled;
+    }
+
+  free (copy);
+}
 
 /* Handle program arguments.  */
 static error_t
@@ -346,6 +402,14 @@ parse_opt (int key, char *arg, struct argp_state *state)
     case OPT_BIG_ENDIAN:
       set_big_endian (true);
       break;
+    case OPT_NO_WARN:
+      /* Disable the warnings.  */
+      set_warnings (arg, false);
+      break;
+    case OPT_WARN:
+      /* Enable the warnings.  */
+      set_warnings (arg, true);
+      break;
     case 'c':
       force_output = 1;
       break;
diff --git a/locale/programs/localedef.h b/locale/programs/localedef.h
index 96aa696..e4f22ce 100644
--- a/locale/programs/localedef.h
+++ b/locale/programs/localedef.h
@@ -115,6 +115,7 @@ struct localedef_t
 
 /* Global variables of the localedef program.  */
 extern int verbose;
+extern bool warn_int_curr_symbol;
 extern const char *repertoire_global;
 extern int max_locarchive_open_retry;
 extern bool no_archive;
diff --git a/localedata/Makefile b/localedata/Makefile
index 75f71bb..a5f3c92 100644
--- a/localedata/Makefile
+++ b/localedata/Makefile
@@ -200,17 +200,25 @@ install-locales: $(INSTALL-SUPPORTED-LOCALES)
 install-locales-dir:
 	$(..)./scripts/mkinstalldirs $(inst_complocaledir)
 
+# The SHIFT_JIS and SHIFT_JISX0213 character maps are not ASCII compatible,
+# therefore we have to use --no-warnings=ascii to disable the ASCII check.
+# See localedata/gen-locale.sh for the same logic.
 $(INSTALL-SUPPORTED-LOCALES): install-locales-dir
 	@locale=`echo $@ | sed -e 's/^install-//'`; \
 	charset=`echo $$locale | sed -e 's,.*/,,'`; \
 	locale=`echo $$locale | sed -e 's,/[^/]*,,'`; \
+	flags="--quiet -c"; \
+	if [ "$$charset" = 'SHIFT_JIS' ] \
+	   || [ "$$charset" = 'SHIFT_JISX0213' ]; then \
+	   flags="$$flags --no-warnings=ascii"; \
+	fi; \
 	echo -n `echo $$locale | sed 's/\([^.\@]*\).*/\1/'`; \
 	echo -n ".$$charset"; \
 	echo -n `echo $$locale | sed 's/\([^\@]*\)\(\@.*\)*/\2/'`; \
 	echo -n '...'; \
 	input=`echo $$locale | sed 's/\([^.]*\)[^@]*\(.*\)/\1\2/'`; \
-	$(LOCALEDEF) --alias-file=../intl/locale.alias \
-		     -i locales/$$input -c -f charmaps/$$charset \
+	$(LOCALEDEF) $$flags --alias-file=../intl/locale.alias \
+		     -i locales/$$input -f charmaps/$$charset \
 		     $(addprefix --prefix=,$(install_root)) $$locale \
 	&& echo ' done'; \
 
diff --git a/localedata/gen-locale.sh b/localedata/gen-locale.sh
index b4ec68c..757a0e9 100644
--- a/localedata/gen-locale.sh
+++ b/localedata/gen-locale.sh
@@ -30,16 +30,12 @@ generate_locale ()
   charmap=$1
   input=$2
   out=$3
+  flags=$4
   ret=0
   ${localedef_before_env} ${run_program_env} I18NPATH=../localedata \
-	${localedef_after_env} --quiet -c -f $charmap -i $input \
+	${localedef_after_env} $flags -f $charmap -i $input \
 	${common_objpfx}localedata/$out || ret=$?
-  # All locales compile fine, except those with SHIFT_JIS charmap
-  # and those fail with exit code 1 because SHIFT_JIS issues a
-  # warning (it is not ASCII compatible).
-  if [ $ret -eq 0 ] \
-     || ( [ $ret -eq 1 ] \
-          && [ "$charmap" = "SHIFT_JIS" ] ); then
+  if [ $ret -eq 0 ]; then
     # The makefile checks the timestamp of the LC_CTYPE file,
     # but localedef won't have touched it if it was able to
     # hard-link it to an existing file.
@@ -57,6 +53,9 @@ charmap=`echo $locfile|sed 's|[^.]*[.]\(.*\)/LC_CTYPE|\1|'`
 
 echo "Generating locale $locale.$charmap: this might take a while..."
 
+# Run quietly and force output.
+flags="--quiet -c"
+
 # For SJIS the charmap is SHIFT_JIS. We just want the locale to have
 # a slightly nicer name instead of using "*.SHIFT_SJIS", but that
 # means we need a mapping here.
@@ -65,4 +64,13 @@ if [ "$charmap" = "SJIS" ]; then
   charmap_real="SHIFT_JIS"
 fi
 
-generate_locale $charmap_real $locale $locale.$charmap
+# In addition to this the SHIFT_JIS character maps are not ASCII
+# compatible so we must use `--no-warnings=ascii' to disable the
+# warning. See localedata/Makefile $(INSTALL-SUPPORTED-LOCALES)
+# for the same logic.
+if [ "$charmap_real" = 'SHIFT_JIS' ] \
+   || [ "$charmpa_real" = 'SHIFT_JISX0213' ]; then
+  flags="$flags --no-warnings=ascii"
+fi
+
+generate_locale $charmap_real $locale $locale.$charmap "$flags"
-- 
2.9.5


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]