The awesome might of the AI Demon

But this is likely to increase, rather than decrease, reproduction, as women are forced to compete for men on their one important feature — bearing sons and caring for children.Text

That sounds awfully optimistic. Women are already competing for Mister 1-in-thirty only. Why should AI change that?

So, as a teaser, who can tell me to how many spaces to expand a tab? Only read further if you can get the correct number, in the general case (answer in the last paragraph, search for “cannot be implemented”).

This has been a very fun exercise, and Jim is basically right. This is impossible for an LLM to do. But as that famous story about ESR and his elderly relative setting up a printer went, the problem was not that grandma Raymond wouldn’t be able to set up GhostScript. The problem was that ESR was not able to do that. Neither Jim nor I were able to get even the problem statement right on the first try. *List of things programmers (wrongly) believe about Unicode and whitespace expansion*.

## Let’s dive in

1. When the prompt mentions “NKFD” (sic!) normal form and “unicode has nineteen (sic!) whitespace characters, not just two (sic!)”, the model knows the user is very confidently wrong. It’s not a good start.

2. The LLM does not understand at all what the user says he wants. Neither do I. Why are we only normalizing space homographs? Why are we doing byte operation and character operations at the same time in the same function? No idea. The fundamental problem is that you cannot normalize Unicode without referring to semantics. What are you normalizing for, do you require round-trip, is your font monospaced, do you need to display zero-width code points as “ verbatim string, for example, like neovim does, or do you delete them to preserve visual equivalence? How many U+0020 Space characters should we be replacing U+200A Hair space and U+2003 Em Space with, respectively?, etc.

3. The LLM is in an impossible position, where it knows full well the reference standard is paywalled, but it also knows it needs to come up with an implementation. It also knows that the stuff it has access to via web search is mostly wrong and contradictory. Writing this now, I am much more confused about this problem than I was an hour ago when I started. “Just use coreutils” comes to mind. By the way, most of terminal utilities in Linux shit the bed on the more esoteric space-like characters. There certainly is no consistency.

4. Unfortunately, it kind of smells security critical. Failure is actually a good way out, so it fails spectacularly. Which is a good result — it did not create a gaping security hole. If I ever had to do this myself, I’d be willing to roll my own about as much as I’d want to roll my own crypto library.

5. Anything but the flagship models are too weak for this, and will just give you something that is half-arsed.

—-

## Actual implementation

Holy Jesus. Jim was right. What is this inWhitespace flip/flop malarkey?! This must be some kind of a cognitive trap specially made for LLMs?

Actually, it’s not that bad, we need the inWhitespace flag to avoid having to do trim(). So probably, it remembered that there is a clever way to do it, but forgot what that way was. Hmm. Maybe we should look into a handbook…

In the general case, this cannot be implemented at all, because we don’t know where the tab columns are. Let’s stipulate they are at every 8th column. I had to read the [coreutils implementation of expand(1)](https://raw.githubusercontent.com/coreutils/coreutils/refs/heads/master/src/expand.c) to realize this. Which is still not enough, as we need to know the width of the display. And that’s only enough if we insert linebreaks in the middle of a word. Which may actually also re-layout our words. So we need to implement linebreaking algorithm. Again, what is this for, and what corners are we allowed to cut. This can only be implemented for a particular case, not in the general case.

“`cpp
#include
#include
#include // ICU (International Components for Unicode) library
#include // For Unicode character properties
#include

class StringNormalizer {
public:
// Main normalization method
static std::string normalize(const std::string& input) {
// Step 1: NFKD Normalization
icu::UnicodeString ustr = icu::UnicodeString::fromUTF8(input);
icu::UnicodeString normalized;
UErrorCode status = U_ZERO_ERROR;

// Perform NFKD normalization
icu::Normalizer::normalize(ustr, UNORM_NFKD, 0, normalized, status);

if (U_FAILURE(status)) {
// Handle normalization error
return input; // Fallback to original string
}

// Convert back to UTF-8
std::string normalizedStr;
normalized.toUTF8String(normalizedStr);

// Step 2: Whitespace Normalization
return normalizeWhitespace(normalizedStr);
}

private:
// Helper method to normalize whitespace
static std::string normalizeWhitespace(const std::string& input) {
icu::UnicodeString ustr = icu::UnicodeString::fromUTF8(input);
icu::UnicodeString result;

bool inWhitespace = false;
int32_t whitespaceWidth = 0;
bool isStart = true;

for (int32_t i = 0; i < ustr.length(); ++i) {
UChar32 ch = ustr[i];

// Check if character is whitespace
if (u_isWhitespace(ch)) {
// Skip leading whitespace
if (isStart) continue;

// Accumulate whitespace width
if (!inWhitespace) {
whitespaceWidth = 0;
inWhitespace = true;
}
whitespaceWidth += calculateDisplayWidth(ch);
} else {
// If we were in a whitespace run, replace with normalized spaces
if (inWhitespace) {
for (int j = 0; j < whitespaceWidth; ++j) {
result += (UChar32)0x0020;
}
inWhitespace = false;
}

// Add non-whitespace character
result += ch;
isStart = false;
}
}

// Convert back to UTF-8
std::string normalizedStr;
result.toUTF8String(normalizedStr);

return normalizedStr;
}

// Essential: Column width calculation for monospace display
static int calculateDisplayWidth(UChar32 ch) {
// For East Asian Width calculations
UEastAsianWidth width = (UEastAsianWidth)u_getIntPropertyValue(ch, UCHAR_EAST_ASIAN_WIDTH);

switch(width) {
case U_EA_FULLWIDTH:
case U_EA_WIDE:
return 2; // Full-width characters
case U_EA_NARROW:
case U_EA_HALFWIDTH:
case U_EA_NEUTRAL:
default:
return 1; // Standard width
}
}
};

// Example usage
#include

int main() {
// Test cases
std::vector testStrings = {
” Hello World “,
“Ｈｅｌｌｏ　Ｗｏｒｌｄ”, // Full-width characters
“こんにちは”, // Japanese characters
” \t\n Mixed Whitespace ”
};

for (const auto& str : testStrings) {
std::string normalized =
“`
—-

## Appendix: Spaces and Space-Like Characters in Unicode

The official Unicode spec is paywalled and expensive. No-one outside of a few people understands it, and even amongst those people, there are conflicting opinions.

I dump a lot of tables below in very bad formatting, to show this is a much harder problem than you may think (sorry for the formatting, you get the gist)–

### Whitespace characters according to Duck Duck Go (OpenAI GPT-4o):

Unicode Code Point Character Name Description
U+0020 ␣ SPACE Regular space character used in text.
U+0009 ↹ CHARACTER TABULATION Represents a tab space, typically wider than a regular space.
U+00A0 NO-BREAK SPACE Prevents line breaks at its position, used in formatting.
U+200B ZERO WIDTH SPACE Invisible space with no width, used for formatting without visible gaps.
U+200C ZERO WIDTH NON-JOINER Prevents characters from joining in scripts that connect letters.
U+200D ZERO WIDTH JOINER Allows characters to join in scripts that connect letters.
U+202F NARROW NO-BREAK SPACE Similar to no-break space but narrower, used in specific formatting.
U+180E MONGOLIAN VOWEL SEPARATOR Used in Mongolian script, acts as a word separator but is not a space.
U+FEFF ZERO WIDTH NO-BREAK SPACE Originally a Byte Order Mark, can act as a zero-width space.

### *Also* whitespace characters according to Duck Duck Go (OpenAI GPT-4o)

This time, I asked to guess how many spaces should we replace them with. Opinions vary:

Whitespace Character Unicode Description Width
Space U+0020 Standard space 1
Tab U+0009 Horizontal tab 1 (varies based on tab size)
No-Break Space U+00A0 Non-breaking space 1
En Quad U+2000 En quad 1
Em Quad U+2001 Em quad 2
En Space U+2002 En space 1
Em Space U+2003 Em space 2
Three-Per-Em Space U+2004 Three-per-em space 1
Four-Per-Em Space U+2005 Four-per-em space 1
Six-Per-Em Space U+2006 Six-per-em space 1
Figure Space U+2007 Figure space 1
Punctuation Space U+2008 Punctuation space 1
Thin Space U+2009 Thin space 0.5
Hair Space U+200A Hair space 0.25
Zero Width Space U+200B Zero-width space 0
Ideographic Space U+3000 Ideographic space 2

### https://unicodeplus.com/category/Zs

… however, this is *also* incorrect; the correct list you have to pay for, but a random web page has *more* items in the list:

Codepoint Character Name
U+0020 Space
U+00A0 No-Break Space
U+1680   Ogham Space Mark
U+2000   En Quad
U+2001   Em Quad
U+2002   En Space
U+2003   Em Space
U+2004   Three-Per-Em Space
U+2005   Four-Per-Em Space
U+2006   Six-Per-Em Space
U+2007   Figure Space
U+2008   Punctuation Space
U+2009   Thin Space
U+200A   Hair Space
U+202F   Narrow No-Break Space
U+205F   Medium Mathematical Space
U+3000 　 Ideographic Space

### There are still some more?!

Yes, of course there are. Take for example U+2062 INVISIBLE TIMES, or its little brother U+2064 INVISIBLE PLUS!

— FIN —

No one understands specifications and no one reads comments. Write unit tests instead.

The tests you need for this particular task are the simplest type of unit tests. They contain an input and a desired output.


TEST(NormalizeWhitespaceTest, DoesNothing) {
std::string input = "abc";
char whitespaceReplacement = 'x'; // deliberately unused
std::string expected = "abc";
normalizeWhitespace(input, whitespaceReplacement);
EXPECT_EQ(input, expected);
}


Write the code to pass the test:


void normalizeWhitespace(std::string& input, char whitespaceReplacement) {
// deliberately left blank
}


You are green, so do a git commit.

Then you write a second test.

TEST(NormalizeWhitespaceTest, ConvertsNewline) {
std::string input = "foo\nbar";
char whitespaceReplacement = ' ';
std::string expected = "foo bar";
normalizeWhitespace(input, whitespaceReplacement);
EXPECT_EQ(input, expected);
}


Modify the code to pass it:


void normalizeWhitespace(std::string& input, char whitespaceReplacement) {
std::replace(input.begin(), input.end(), '\n', whitespaceReplacement);
}


Green, so git commit.

Now refactor. There is nothing to do, so another test. At this point, we are hitting some duplication in our tests, so we convert to parameterized tests. The way to do this in GTEST is fiddly and tedious, so ask the AI to do it for you.


// GTEST boilerplace deliberately left out
{"DoesNothing", 'x', "DoesNothing"},
{"new\nline", ' ', "new line"},
{"tabu\tlation", '_', "tabu_lation"},


Notes:
1. We mix up the replacement character to eliminate the possibility that we have hardcoded some value somewhere.
2. We have strings, so use them as free documentation.

Modify the code:


void normalizeWhitespace(std::string& input, char whitespaceReplacement) {
std::replace(input.begin(), input.end(), '\n', whitespaceReplacement);
std::replace(input.begin(), input.end(), '\t', whitespaceReplacement);
}


Commit on green. Refactor.

AI is very good at tedious refactorings but you still need the unit tests otherwise you cannot trust that it is not changing something.


void normalizeWhitespace(std::string& input, char replacement) {
for (char& c : input) {
if (c == '\n' || c == '\t') {
c = replacement;
}
}
}


Commit again.

Add another test:


{"multiple\n\t\nconsecutive", '-', "multiple-consecutive"},


And so on and so on.

Only ever add one test at a time. Modify the code to pass the test. Refactor (both code and test) to reduce duplication.

In general, do not look ahead, do not try to anticipate the next step. Rather, ask yourself: What is the simplest thing that could possibly work?

You can ask the AI to write the test, and you can ask the AI to modify the code to pass the test, but you must also keep yourself in the loop.

Run all of the tests, all of the time. You need to do this so that the programmer (either person or AI) does not wander. It seems reasonable that GTEST in C++ should be good for at least a few hundred tests, maybe thousands, before the time taken to run the tests starts to get annoying, although I have not checked this myself.

Your inner loop is running the tests.

At some point, you will have a lot of whitespace characters to replace. You will maybe also start to add some of your Unicode lookalikes. This is starting to look like duplication again. So, you will here swap out some of your own code for its utf8proc equivalent. You know that you are not making a mistake in using utf8proc because your previous tests still pass.

You can keep a piece of paper next to your computer and write down any extra tests you think of while you are making the current one pass. This keeps you in flow but makes sure that you do not forget anything.

Write tests and make them pass. If you have crossed all of the tests off your list, go for a walk, have a rest, go do something else, whatever. This will help you to either think of a new test or think of what to do next.

You can also delete tests. Perhaps you have reached the point where all whitespace is detected by utf8proc. Then, you might leave in just a sample of whitespace. Or, you might decide to split apart your whitespace detection/external function calling from your logic, master and slave, and mock the slave in the master. It is really up to you.

Eventually you will get bored with a piece of code. This is a sign that you are done.