Handling international characters with the Mixed Encoding Scanner and data converter

Character encoding tells databases how to handle data. For example, when you’re using the Quickbase platform, Quickbase sends your data to a browser and tells it what character set the data is so the browser can display it correctly on the screen.

The UTF-8 character set is the universal standard for encoding international and US English characters. By default, UTF-8 encoding is switched on for Quickbase accounts created after 2022, but admins can choose to turn it off if desired. For accounts created prior to 2022, admins turn this setting on for their apps and realms. Learn more about UTF-8 settings for Quickbase apps and realms.

What is mixed character encoding?

If non-English characters were entered into your Quickbase apps before UTF-8 support was added or before you activated it in your apps, you may end up with mixed character encoding. It’s also possible that mixed encoding resulted from imported or copied data. This can lead to inconsistencies with displaying, sorting, filtering, and processing data.

Here’s a little background on how it works. If non-English characters were entered into an app where the UTF-8 switch is turned off, the characters may still display correctly to the person entering the data at the time because you can enter any text string into a text field—including UTF-8 characters—and not receive an error. However, if you don’t tell the system to store them as UTF-8 characters, they might be stored in a different encoding that doesn’t support all characters or all languages. This can result in issues retrieving and displaying the data.

Simply turning the UTF-8 switch on won’t solve this problem because Quickbase will not automatically re-encode existing data. Quickbase will be able to correctly identify UTF-8 characters when sending data to a browser. However, when UTF-8 characters aren’t encoded correctly, a browser won't know how to handle them. When this happens, you might see characters displayed as “?” or bytecode, which looks like this—#00FB.

Using the Mixed Encoding Scanner app to find mixed character encoding

If you think you might have mixed encoding, the Mixed Encoding Scanner app can scan your data for you. It scans for UTF-8 data and will identify any tables that contain it. If it finds instances of mixed encoding, you can easily convert your data afterward.

To get a copy of the app, click Explore Sample Apps on the My Apps page and search for "Mixed Encoding Scanner."

After getting the app from Exchange, you can analyze any app in your realm. Enter the app ID, also called the database ID (DBID), and an app token to get started.

Next, the Scanner will estimate how long the analysis will take to run. When the analysis is complete, you can browse the results to determine which tables, fields, records, and app variables need to be fixed.

How to resolve mixed encoding issues

Resolving mixed encoding will correct issues with storing, retrieving, or processing your data.

If the mixed encoding scanner returns only a few occurrences of mixed character encoding, turn on UTF-8 encoding for the app, and then manually edit improperly encoded data. Simply re-type the characters that aren't displaying properly.

Another way to convert improperly encoded data is with the encoding conversion tool. Use this tool to convert:

• Text (Standard, multi-line, multi-choice, rich text, and composite parts of an address field)
• Text - Multi choice 
• Multi select 
• URL 
• Phone 
• Email 
• Schema
• Table name 
• Field name 
• Field help text 
• Field label on legacy forms 
• Report name 
• Report description 
• Field choices in a multi choice 
• Currency symbol 
• Form name 
• Link text on a formula URL
• Comments on a field 
• Table description 
• Section names and tab names on legacy forms 

Using the encoding converter

In your app, click App management → Manage App Contents → Encoding conversion. Then, you can adjust the settings for the conversion. Choose the encoding types you'd like to convert by selecting them from the drop-down menus labeled From and to. Click Convert.

 You'll be prompted to confirm the data conversion again in a dialog box.

If data was converted, you'll see a confirmation message and a report of what was converted.

If nothing was converted, you’ll receive a message that says there was no data to convert.

Good to know:

• While this tool is equipped to process large amounts of data, it can’t guarantee 100% of data will be converted. Run the mixed encoding scanner afterward and manually correct anything that the converter didn’t catch.
• Converting data from Win-1252 with automatically turn UTF-8 encoding on for that app.
• You can go backwards, or convert data from UTF-8 to Win-1252 if you FUBAR (Fast, Unconstrained Bayesian AppRoximation) your data.
• If you lose any data during conversion, Quickbase can restore it for a limited time. You can also manually back up your data.

Additional resources

• If you’d like assistance with problematic characters or mixed encoding in your app, you might consider working with Data Collaborative.
• Advanced builders can choose to leverage the encoding techniques using VSCode or a similar tool to handle mixed character encoding.