Make whitespace invalid in token names

[ChucKN0risK]

We've decided to consider a token as invalid when its name includes whitespaces.

1. I've added a mention of this in the design-token.md file
2. I've transformed to kebabcase all token names used in the spec in all pages to make them respect this new decision

[netlify]

✅ Deploy Preview for dtcg-tr ready!

Name	Link
🔨 Latest commit	1b494d4
🔍 Latest deploy log	https://app.netlify.com/sites/dtcg-tr/deploys/643fb09d143c160008830366
😎 Deploy Preview	https://deploy-preview-216--dtcg-tr.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[romainmenke]

This is a major reversal of past choices.
Can you link to any notes/logs where this was discussed?

[dhwlz]

Suggested change

	"Token-cuatro": {
	"token-cuatro": {

[dhwlz]

Suggested change

	The name of your design token must not include any whitespace. Otherwise your design token is considered as invalid.
	Design tokens are written in kebab-case. Otherwise your design token is considered as invalid.

[c1rrus]

I think this should be left as is. Kebab-case is only one way of avoiding whitespace... snake_case or CamelCase are fine too.

[drwpow]

Sorry for pinging an old PR; was just reviewing the open ones. If we want to ban whitespace, should we do something like say adopt CSS’ identifier syntax? Where foo bar isn’t allowed, but foo\ bar is (escaping)? I’d like to have a complete specification on which characters are/aren’t allowed just so we’re clear (and we can throw errors better)

Of course, this would get weird in JSON where I think foo\\ bar would be required 🤔 which would interfere with readability.

This is a major reversal of past choices.
Can you link to any notes/logs where this was discussed?

+1 to this question—what problems were we running into where whitespace was the problem? I’m not necessarily opposed to the idea, just lacking context that I can’t find in this PR

[kaelig]

White space makes it harder to write aliases, but aside from that, I'm not sure what the impetus was. @ChucKN0risK can you write down an explainer of the pros and cons for this approach?

[c1rrus]

@kaelig & @drwpow from what I recall, (part of?) the intent was to avoid naming clashes when token names are exported as code by translation tools (e.g. Style Dictionary, Terazzo, etc.).

In JSON property names are case sensitive and may contain whitespace. So, you could have valid DTCG file like this:

{
  "My token": { "$value": /* ... */ },
  "My   token": { "$value": /* ... */ },
  "my-token": { "$value": /* ... */ },
  "   my     token    ": { "$value": /* ... */ }
}

...and those would be 4 different tokens.

However, a lot of programming languages have more restrictions and/or naming conventions for variable names , so when exporting to code, those token names will often end up being kebab-cased, or camelCased, or have some other kind of transform applied to them.

Unfortunately, that can sometimes lead to different tokens names that are valid in JSON/DTCG turning into the same output name. E.g. camelCasing the example above, might product the same myToken name for all 4 design tokens in the above example.

Also, if I'm not mistaken, in JSON a property name could be entirely whitespace or even an empty string. So this would technically be valid too:

{
  " ": { "$value": /* ... */ },
  "": { "$value": /* ... */ }
}

How would you export that to a JS variable? Or, display as a Figma variable name?

I think there was a suggestion that perhaps, tools should treat token and group names as case-insensitive and ignore whitespace. So, " my token ", "mytoken", and "MyToken" would all be considered equivalent. And then, if the were duplicate normalised names within the same group, that should be considered an error. So, then the 2 tokens in this example:

{
  "My token": { "$value": /* ... */ },
  "My   token": { "$value": /* ... */ },
}

...might have their names normalised by lowercasing and removing whitespace, which could lead to mytoken in both cases and should then be considered an error by a DTCG parser. However, the concern was that this would create more complexity for DTCG parsers. Especially, if you consder references too:

{
  "My token": { "$value": /* ... */ },
  "alias token 1": { "$value": "{mytoken}" },
  "alias token 2": { "$value": "{my-token}" },
}

...should both alias tokens be valid and point to "My token"?

So, I think the idea was, it might be easier to impose limitations on the names of tokens (and groups) in DTCG file to avoid needing to deal with those kinds of edge cases and complexities.

@ChucKN0risK and @dbanksdesign may recall more details and be able to provide additional rationale.

Personally, I've always been on the fence about this one. Looking at it with fresh eyes now, I do wonder if this is something we can't just leave as is, or perhaps just impose fewer restrictions. Given that there's now quite a few DTCG tools and files out there in the wild, this could be quite a big breaking change for folks. I know the spec is still an editor's draft, but I'd argue it's become a kind de facto standard already, so we should tread carefully.

Perhaps names that are only whitespace should be forbidden, since they're difficult to display in a meaningful way in a GUI (e.g. in a design tool) or export to code. I doubt anyone would miss the ability to name a design token " ".

There'd still be the potential risk of naming clashes when exporting to code, but I think that's something that export tools could detect and warn/error on (I think Style Dictionary already does this). Also, we could include a recommendation in the spec that file authors avoid choosing sibling token/group names that only differ in case, whitespace or separator characters like - and _, which are likely to be removed or converted when exporting to code. Tools that let you create or edit DTCG files could also discourage or prevent that kind of thing.

[drwpow]

@c1rrus That all makes sense! Thanks for all the explanation and examples. I agree in providing guidance for these scenarios. Also agree surrounding whitespace should be invalid.

I wonder if we should have a more complete identifier specification, similar to CSS ident or something. Rather than incremental changes to disallowed characters.

[c1rrus]

Yeah, I agree. Something along the lines of CSS ident would be good.

Should we maybe reject this PR and propose something along those lines + guidance in a new PR?

[ChucKN0risK]

@kaelig @drwpow I've not been able to find any information in our meeting summaries about this change. But from what you've described @c1rrus I fully agree with you. Let's let token names include whitespaces while preventing a name from just being whitespaces.

I can create a new PR along the lines of CSS Indent.

Just please confirm me we all reached an agreement here 👍

[romainmenke]

CSS idents do allow any character as long as it is escaped.
Either through a backslash or by using a literal code point.

I assume you mean to constrain token names to what is valid in CSS idents without any escaping. So fo\ o would not be a valid token name even when it is a valid CSS ident?

[ilikescience]

I'd propose instead of standardizing on CSS idents in this case, we use the JSON spec (specifically, strings, which are the only allowed type in a key) as a foundation for token names; there are valid css idents that are not valid JSON strings*, so if a dtcg file was evaluated as json then some funky things might happen.

* \a is a valid css ident, but not a valid JSON string.

[d34dman]

I'd propose instead of standardizing on CSS idents in this case, we use the JSON spec...

+1 on this one.

As far as generating and managing DTCG file by humans, simple rules for managing spaces has great value.

However, the token name is an identifier that has a far reaching significance for tooling. Unlike CSS , which targets one target system (CSS parsers), the token name for Design token act as an identifier not just within the JSON document that represents Token (or list of tokens), but via translation tool with other ecosystems as well (It could be a CSS variable, Class name, Attribute name, ...).

It would be great usability improvement, if a token name is still "identifiable uniquely" after it is translated. This goes a long way in developing reliable tooling and predictable behavior for references and aliases. Which in turn, makes human life easier when debugging design token related issues in target implementation.

Proposoal,

Following is modified CSS spec, after dropping support for escaped character.

• any ASCII character in the ranges A-Z and a-z
• any decimal digit (0 to 9)
• a hyphen (-)
• an underscore (_)
• any other Unicode character U+00A0 and higher (that is, any other non-ASCII Unicode character)

[phun-ky]

@d34dman hm, so, with that spec, we would allow emojis? 🤔 we could look at the "key" in JS objects as a leading guide here IMHO. Or am I way off?

[d34dman]

@d34dman hm, so, with that spec, we would allow emojis?

The idea was not about allowing emojis, but to maintain the support for non latin scripts. Emoji support was a side-effect of that.

Or am I way off?

The proposal is different in a way that, we specify an allowed list of characters for token, instead of a dis-allow list.

JS Object's key are strings right? So then, we will need to dis-allow some characters like [".", "{", "}"], and make the list evolve into spaces and some more ¯\_(ツ)_/¯ perhaps.

We might be better off by declaring which values are valid, instead of keeping a list of invalid characters. This way we can introduce more control characters if needed, with backward compatibility.

E.g. A different reference format using characters like "#" and "/" as discussed here #259 or adding more control characters like "+" or "&" for predictable mixing and composing of tokens. At this point this is mere speculation though.

[ilikescience]

I'm still a fan of the JSON definition of a string, as it is:

1. very precise, and
2. well-supported from a tooling perspective

Here's the definition:

The upside is that we can say a DTS file must be valid JSON. On the downside, it does mean that whitespace is valid in token names. We can follow up on what is syntactically valid vs. recommended/not recommended as a non-normative thing.