IBM Developer

Article

i18n for fun and profits for all of your Golang programs

Use the i18n4go toolset for internationalization

By Nana Amfo, Michael Maximilien

On this page

Internationalization (i18n) is an important part of any software. While delegated to secondary feature status, i18n is mandated by various governments around the world, and the United States (Section 508 Compliant Translation and Localization Services) requires that organizations make their software available in a variety of languages. Because Golang is increasingly used to create cloud-based software for both the back end and the front end, there is increasingly a need to create an internationalized version of a Golang program.

i18n4go was created in 2015 as a toolset to help internationalize Golang programs. Its first and largest use case is the Cloud Foundry (CF) command-line interface (CLI). The CF CLI was used to create and improve i184go and to discover its main use cases and user processes.

In particular, i18n4go is designed to help internationalize existing and new Golang programs without changing the source. In other words, you can use i18n4go to create an internalized version while keeping the existing program non-internationalized. This article explains the details of how to use i18n4go as well as the new features since the program was created in 2015.

i18n problem

At its core, the i18n problem is the ability for a program to communicate to its human users in their native language. This means that all of the strings that make up the program’s user interface can be in the native language of the human user. This includes active and passive UI components, from input text to dialog boxes to logging strings. Additionally, the concept of localization (i10n) is the ability of a program to expose information in a localized fashion; for example, showing the data as DD/MM/YY versus MM/DD/YY depending on the country (or local) norms. i18n4go was designed to help address the i18n problem, not i10n problems, although you can use i18n4go to help when localizing a Golang program.

Motivation

As Golang gained popularity as the primary language for cloud computing development, so did its use for CLIs. Because CLIs are meant for end users, there is a need for internationalization of these CLIs. This requirement became an important issue for Cloud Foundry, and, specifically, the IBM Cloud (previously known as Bluemix) release of Cloud Foundry in late 2015. The i18n4go toolset was created to address this Cloud Foundry CLI need and was generalized as a toolset after it was used in the Cloud Foundry CLI.

Golang support

Currently, Golang does not support a native package for international translations. However, Google has provided a package that interfaces with the Google Cloud Translate SDK for translating messages. The following table shows a list of the features against its competitors. i18n4go uses the go-i18n framework.

Package Language detection String formatting Offline support
go-i18n Yes Yes Yes
google-cloud-go/translate Yes No No

Why i18n4go?

The initial goals and features of i18n4go were to support the conversion of Golang programs (for example, CLIs) into internationalized programs. This means supporting a variety of spoken languages in both input and output as well as supporting the process of converting and managing the internationalization of a Golang program. This section describes a brief history, use cases, and users of the toolset.

History

The Cloud Foundry CLI was the primary way that users of Cloud Foundry used and managed applications on that platform. When creating Cloud Foundry, it became clear that the CLI needed to be internationalized before the platform could be used worldwide.

Because the Cloud Foundry CLI included dozens of commands with dozens of unique strings each, the process of converting the CLI needed to be automated. This was the only way to convert the current CLI while making it easy to add new commands and new strings to the system.

With this initial constraint, we engaged with the Cloud Foundry CLI team to create a toolset that could facilitate this conversion. The outcome of the effort became the i18n4go toolset.

Use cases

18n4go has many common use cases. However, this article only discusses the two that we have seen used in the open source communities and at IBM.

  1. As a way to convert existing Golang code and make it internationalized
  2. Using the tool to maintain internationalization files

The first use case, as a way to convert existing Golang code and make it internationalized, is the primary scenario that led to the creation of this project. In essence, this use case takes full advantage of the tool.

You start with a non-internationalized Golang project and with various runs of i18n4go, and you end up with new rewritten code that is internationalized. You provide the translations (one per language) and rebuild your code to create a binary that embeds the translations, and, therefore, can use the internationalized strings. This is precisely how the Cloud Foundry CLI project uses the i18n4go project.

A second use case is to use the tool to maintain internationalization files. Because the tool has various independent commands, you can use it to extract strings in a Golang program and to check that strings in a translation file are present in the Golang program.

Latest release

When it became time to create a new version of 18n4go, the new version was designed to address two primary changes to the project. First, it fixed the command structure by removing the unnecessary -c flag to indicate command. This should have been done from the beginning, but it required a fair amount of changes and the use of a new Golang command library.

Second, the new version streamlines the project with better CI/CD and automates things as much as possible. The release of the Cobra library along with GitHub actions have created the perfect time to implement these new features in release i18n4go v2.

New features

The new features in i18n4go v2:

  • Rework the entire command structure to use subcommands without using the -c to execute the subcommand. For example:

          i18n4go -c checkup [old]
      i18n4go checkup [new]

  • Provide backwards compatibility with the -c command flag

  • Upgrades the nicksnyder/go-i18n package to the latest version and:
    • Provides security fixes
    • Improves locale detection using the Golang build language package
  • Provide updates to the latest version of Golang (currently, v1.20.0)
  • Add improvement with detecting localized strings that are initialized through variables when rewriting packages
  • Add the optional --ignore-regexp flag to let users exclude files and folders when checking localization files against source code when using the checkup command
  • Provide localized i18n4go source code using the i18n4go command

A quick user’s guide

Let's say that you have a CLI program and you want to add localization to your error and log messages. Using i18n4go, you can quickly generate language files with a few commands from the terminal.

Before beginning, it’s a good practice to place your localization files in a directory that’s separate from the main source files. In this guide, we place the localization files under the app/cmd/i18n/resources directory. The guide references source code that can be found in these GitHub files. You can download and unzip the files to follow along.

Also, when running commands, make sure to run them at the root of the project.

  1. Open the root.go file, and determine which strings should not be translated. In the previous examples, you can exclude "", version, and v because these strings do not change regardless of language locale.

  2. Create a file called excluded.json in the resources directory. To exclude the strings using regular expressions, add the strings to the excludedRegexps array. Otherwise, add them to the excludedStrings array.

     // app/i18n/excluded.json
 {
     "excludedStrings": [
         "",
         "version",
         "locale",
         "en_US",
         "foo",
         "foo <arg>"
     ],
     "excludedRegexps": [
         "^[a-z]{1}", // matches all single characters flags (e.g. 'v', 'l')
         "^v([0-9){1}\\.([0-9]){1}\\.([0-9]){1}" // match versions (eg. v0.0.1)
     ]
 }

  3. Run the i18n4go extract-strings command to extract the strings from root.go into a JSON file. If you have multiple files that contain translatable strings, the command generates a localization file per source file.

     i18n4go extract-strings --directory app/cmd --recursive --output app/i18n/resources/ --exclude app/i18n/excluded.json

    A localization JSON file named root.go.en.json is generated under the app/i18n/resourcesfolder with the strings in the excluded.json file not added. By default, the source language is the en_US locale.

     // app/i18n/resources/root.go.en.json
 [
     {
         "id": "Test cli command",
         "translation": "Test cli command"
     },
     ...
 ]

    The i18n package library used by i18n4go expects the localization files to be named in a specific format. The format is all.<locale>.json. You can run the merge-strings command, which can handle the renaming for you. This command is useful if you have multiple generated localization files.

     i18n4go merge-strings -–directory app/i18n/resources/ —–source-language en

    A localization JSON file named all.en.json should have been created under the app/i18n/resources folder.

  4. Rewrite the source file (root.go) to use the T(...) method to translate the strings when needed. Before you run the command, you must first do a few things.

    1. Generate a binary file that contains your localizations. Using the binary file makes translation faster because the translation is loaded in memory instead of directly from the JSON file.
    2. Create a template file called i18n_init.go.template that is used in generating the i18n_init.go file. You can use the i18n_init.go.template reference in the previous demo project. This file is responsible for defining the T(...) used through the source code.

Generating a binary file containing your localizations

It’s recommended that you use go-bindata to generate your binary file.

go install github.com/go-bindata/go-bindata/v3/go-bindata@v3.1.3

After you have installed go-bindata, generate your binary file containing your resources. Run the command assuming that you are at the root of the project.

go-bindata -pkg resources -o app/i18n/i18n_resources.go -nocompress app/i18n/resources/*.json

Check and see that the app/i18n/i18n_resources.go file has been created.

After you have generated the i18n_resources.go file, you can rewrite the source code that will wrap T(...) around your localized strings, where appropriate. The T(...) method is defined in the template file.

i18n4go rewrite-package --verbose --directory app/cmd --i18n-strings-filename app/i18n.resources/all.en.json --init-code-snippet-filename i18n_init.go.template

After running the command, you should see an output similar to the following one that describes what has changed.

i18n4go: rewriting strings in dir app/i18n, recursive: false

i18n4go: loading JSON strings from file: app/i18n/resources/all.en.json
i18n4go: rewriting strings for source file: app/i18n/i18n_resources.go
i18n4go: using the PWD as the rootPath: /home/noamfo/Documents/i18n-demo
i18n4go: determining import path using root path: /home/noamfo/Documents/i18n-demo
i18n4go: got a local import . so using /home/noamfo/Documents/i18n-demo instead for root pkgi18n4go: got a local import . so using /home/noamfo/Documents/i18n-demo/app/i18n instead for pkgi18n4go: using import path as: app/i18n
i18n4go: adding init func to package: resources to output dir: app/i18n
i18n4go: inserting T() calls for strings that need to be translated
saving file to path app/i18n/i18n_resources.go

Total files parsed: 1
Total rewritten strings: 0

Open the root.go file, and verify that the T(...) method wraps your localized strings.

package main

import (
    "fmt"
    "os"
    "github.com/spf13/cobra"
)

const VERSION = "v0.0.1"

type cmdOptions struct {
    version bool
    locale string
}

func main() {
    opts := cmdOptions{}
    rootCmd := &cobra.Command{
        Use: "foo",
        Short: T("Test cli command"),
        Long: "foo <arg>",
        Run: func(cmd *cobra.Command, args []string) {
            if opts.version {
                fmt.Printf(T("Version {{.Version}}", map[string]interface{}{"Version": VERSION}))
            }
    },
}
...

The localization is not useful with only one language. You can create new translations by using the create-translation command. By default, the command creates new localization files by duplicating the source localization file (for example, en_US.json). You can provide a Google AppEngine API key through the google-translate-api-key optional flag that will use the Google Translate API to help generate your translation. Be aware that this might incur charges on your Google AppEngine account if used.

The following command is an example of creating translation for fr_FR and es locales.

i18n4go create-translations --file app/i18n/resources/all.en.json --languages fr_FR, es --output app/i18n/resources/

After you are satisfied with the contents of your localizations files, you can verify that there are no missing translations across the different languages. Run the verify-strings command.

i18n4go verify-strings --file /path/to/resources/source_translation.json --languages fr_FR, es

Besides checking for missing translations, you can also check to see if all of your translations are being used in your code. Run the checkup command.

i18n4go checkup

Conclusion

This article provided a complete update on the i18n4go toolset. This is a Golang tool suite that you can use to convert and update your Golang program so that they can be internationalized. This toolset was updated to version 2, with some significant internal changes. Externally, the usage is the same.

What's next?

A big part of any open source project is the vibrancy of its community. Without a vibrant community, an open source software project cannot survive nor thrive. While i18n4go is a small project, early on it had a good community that helped it grow. However, the community has dwindled to a few developers and companies actively using it. Building back the community to have diverse users and developers is the top priority moving forward.

Additionally, another top priority moving forward is to align the project with the latest versions of Golang. Besides addressing any bugs and issues that are discovered since the v2 release, the priorities in terms of development centers around the twin issues of first aligning with the new i18n packages in Golang and taking advantage of new Golang language features.

We believe that taking advantage of the new i18n package released by Google could remove the dependency of goi18n. And since Golang 1.18, new features like generics and built-in localization might help facilitate current i18n4go packages that have thorny code that is difficult to maintain. However, we do not want to attempt these refactoring changes without having a better idea if they would indeed help. This is why we plan to do them as a work-in-progress while focusing on bug fixes and growing the community.

Acknowledgment

Max would like to acknowledge Nana Amfo for being the main developer for version 2 of i18n4go. During 2023, Nana worked diligently and tirelessly to update, fix, and discuss the various issues and code changes needed to create v2, which is the biggest release of the project since its inception.