There are an excessive amount of programs that conflate key presses & text input, and ones that don’t consider input methods.

I use macOS & Linux, and while the default text handling system called Cocoa Text System in macOS handles input methods well, almost all applications that implement it’s own, like big apps like Eclipse and Firefox, don’t get this right.

On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally, and after a week of use you get used to pressing space & backspace after finishing every Hangul word. The Unix-style composability they want (apps should work whether or not input methods are used - and looks like Linux users that use Latin characters don’t use any input methods (opposed to macOS where Latin characters are input by a Latin input system), so looks like this state will persist.

About the emoticons, I’m not that concerned with that since most (if not all) users won’t really input the color modifier separately (or even encounter files that have a separate one), so you can just select a sensible behavior like the #2 or 3 or 4. Users who understand the color modifiers, and other Unicode fiasco will understand what is happening under the hood, and ones that don’t will just think the file is broken and none of the behaviors will make sense, whatever you do.

Of course, everyone's preferences will be different, so you'll get default IMEs with default configs, but with the idea that you can reconfigure or replace them entirely with systems that work the way you want to work everywhere you input text. There have been utilities that do this sort of things for decades, but they've always been treated as clever hacks rather than standard text input.

In other words, instead of powerful text input methods being the exception, they would be the rule, and apps that didn't use them would be the exception.

This is so true, all commercial OSes that take i18n seriously do this, while most open source OSes' community (which communication revolves around English) decided that IMEs are an add-on for CJK people.

It's a pity, and this one reason is enough for Linux to be never adopted for ordinary users in the non-western world.

My biggest problem with IMEs in free software land is that we've had groups like Gnome lean on IME developers and push through their vision of how it should work -- even though the people pushing their vision don't use IMEs. I ended up migrating to FCITX just because it was the last hold out not to cave to pressure.

Microsoft and Apple have the incentive of selling to billion-user markets. The open source community, on average, appears to have demonstrated a lack of interest in opening up the user-base further (and expecting that user-base to just roll their own solution creates multiple catch-22 and tragedy of the commons problems).

Why one of the commercial open-source vendors hasn't taken this on as a core challenge, I do not know.

Technically true, but for practical purposes IBus is the standard. That's what Fedora and Ubuntu use out of the box. Firefox beta telemetry shows about 89% IBus vs. 11% FCITX.

> My biggest problem with IMEs in free software land is that we've had groups like Gnome lean on IME developers and push through their vision of how it should work -- even though the people pushing their vision don't use IMEs. I ended up migrating to FCITX just because it was the last hold out not to cave to pressure.

What wrong thing did the Gnome folks push for?

Some things evolved, some things didn't. O hi X Windows!

They are all very nearly the same age.

Windows 1.0 was fairly primitive, but Windows 2.0 supported overlapping windows (!) in 1987, coincidentally the year that X11 was released.

Does Windows really?

> while most open source OSes' community (which communication revolves around English) decided that IMEs are an add-on for CJK people.

IIRC, Fedora ships an iOS/Android like Latin-script IME with autocomplete. It's not mandatory, though.

> It's a pity, and this one reason is enough for Linux to be never adopted for ordinary users in the non-western world.

The Korean situation doesn't really generalize. Among CJK, AFAICT, the Korean IBus IME on Ubuntu 18.04 is pretty broken but the Japanese IME and the various Chinese IMEs appear to be at least OK. (It's a rather surprising situation considering that the Hangul part of a Korean IME should be much simpler than the Japanese and Chinese IMEs.)

Yes, the signs that even English is passing through Windows' IME infrastructure in Windows 10 is pretty minimal by default (in Desktop mode, Tablet mode immediately turns on a couple more), but at this point Windows 10 makes almost all of it opt-in. Some of it is referred to as accessibility tools from an English perspective, because IMEs are also useful for accessibility.

The "big one" IME for most English users is the Emoji Keyboard accessible with Win+. or Win+; (whichever you prefer). It's really interesting how well emoji have helped Latin script users with further understanding the complexities of Unicode, fixing old ugly bugs in Unicode handling, and even introducing some such users to an IME that they want to use (sometimes every day).

Under Settings > Devices > Typing > Hardware Keyboard you can turn on the IME "Show text suggestions as I type" even on a hardware keyboard in Windows 10, which gives you mobile-style auto-suggestions (you can also turn on mobile-style autocorrect even on a hardware keyboard).

No, API-wise the on-screen keyboard generates keystrokes for emoji (astral keystokes, multiple keystokes for multi-scalar-value emoji) even to an IME-aware app. In contrast, the emoji picker built into the Windows 10 Pinyin IME enters emoji via IME API.

> Under Settings > Devices > Typing > Hardware Keyboard you can turn on the IME "Show text suggestions as I type" even on a hardware keyboard in Windows 10, which gives you mobile-style auto-suggestions (you can also turn on mobile-style autocorrect even on a hardware keyboard).

Thanks!

That said, even if it isn't using the IME APIs directly in most cases, it's still useful as a teaching tool/analogy tool/example tool to English speakers of what an IME can be like to use, even if a nice-to-have for an English writer versus a necessary required tool for other languages.

Windows (10 at least) has native support for these IMEs; setting it up on Windows was much easier than on Linux (which doesn't even have a standard IME). Windows also comes with basic CJK fonts, which different distros may or may not have. On Debian I have to install noto-cjk or Adobe's source han fonts.

I originally tried fcitx but had to switch to ibus. Definitely not a great experience.

In my experience, Debian is worse than Fedora, Ubuntu, and openSUSE. Fedora is better than Windows 10 when it comes to IMEs: Fedora installs the IMEs by default. Ubuntu, like Windows 10, installs IMEs when you request the addition of an IME-requiring language. OpenSUSE gives you an IME for the language you use at install time if you install openSUSE using an IME-requiring language.

I haven't tried Debian 10, but when I installed Debian 9 _in Japanese_ with a _Japanese keyboard layout_ chosen, the installer didn't bother to set up a Japanese IME!

Fedora comes with an OK set of Noto CJK fonts by default. Ubuntu comes with a minimal set of Noto CJK fonts by default, but when enabling Chinese, Japanese, or Korean, Ubuntu drops more language-appropriate fonts on the system, like Windows 10.

Which is why it's infuriating when the built-in widgets of a platform don't include simple operations for obvious workflows like filtering characters, masked input, etc. That's what leads developers to have to roll their own implementation using keydown/keyup.

I can't tell you how many GUI frameworks have forced me to roll my own numeric input widget.

I have a note-taking system that I use to store all this data, and after I set it up years ago I looked at implementing an IME to search my notes and get a link to an entry, but every platform has its own byzantine IME API and I never got anything working. I've been thinking it might be easier to just make a cross-platform program that interacts with the clipboard rather than being a true IME.

In general one effect I'd like to see from having flexible IMEs would be allowing logging and chat could be left to separate apps. Imagine a birdwatching (or whatever) keyboard that lets you pick the birds you saw, their age, behavior, and punch it in while you're in the field, and have it output in structured form into whatever app is handy.

I can't agree here (excepting perhaps the unicode char replacement which a non-english speaker needs to comment on the viability of, vs actually having a non-english keyboard (excepting huge alphabets like chinese where that is used already AFAIK)).

Writing is fundamentally a thinking process, the text entry for a touch typist is relatively quick. If you are to propose expanding abbreviations, how much time do you expect to save? I mean, actually measured it?

> runs little scripts and inserts the output

What is the purpose of this?

> gives you quick dictionary/thesaurus lookups

I use those perhaps once a week or less. If you use it say 3X a day, you'd save little time, perhaps a couple of minutes.

Dunno what emmet powers are though.

MS word & libreOffice does some of what you want and the first time I install them, I spend several minutes tracking down each setting and turning them off - they drive me bonkers. They think they know what I want but they don't. Touch typists can hit many keys a second and kind of pipeline their typing. Having the input modified automatically is rarely useful IMO.

Your idea may be good but like many other ideas such as graphical programming, except in restricted cases they don't work. Perhaps if you measured it I'd be convinced but I can't accept it now as an obviously great benefit.

Not quick enough. I think faster than I type, and I type fast. It's fine until I start getting impatient with myself. Call it micro-impatience, a flash of irritation in which you're suddenly conscious of not having finished typing in the thought. It's distracting.

Honestly, I like parent's idea; a good chunk of Emacs's awesomeness and the reason people like me use it as an operating system is because of that - unified, fully configurable and expandable text-based interface. I often wish to have something like it system-wide, because standard UIs are very far from optimum ergonomy-wise. But then again I wouldn't trust Apple or Microsoft to do it right; they'd quickly find a way to dumb it down, or restrict the extensibility in the name of security.

I can totally imagine that. Underneath all the GUI layers, every operating system and application has (or has the potential of) a fully text-based interface. There's just no standard or integration, and tools that allow that (like a system-wide middleware) haven't caught on, I guess. Maybe in an alternate historical timeline, such a feature could have been a fundamental layer of an OS.

From the grandparent comment:

> a system [with powerful text input methods] that expands abbreviations, replaces easy-to-type sequences with proper Unicode chars, runs little scripts and inserts the output, gives you quick dictionary/thesaurus lookups, gives you emmet-style powers, etc., whenever you're writing and in any app.

Yes, yes - and the last point: in any app. I picture it like how TCL can script other programs, even ones that weren't designed to be "remote controlled".

Why do you say this? There is nothing fundamental about text. Is there something fundamental about text in Smalltalk? Or AmigaOS?

> There's just no standard or integration, and tools that allow that (like a system-wide middleware)

COM on Windows. Scripting interface to apps on MacOS. They're there.

This is a reality within Emacs (which really is a 2D text OS running lots of applications inside, including a text editor), and being text-based does play a role. When it's very hard to draw arbitrary pixels on screen and most of all apps deal with text, it's easy to make a large set of very powerful interface tools, and it's easy to pull data out of an app and put data into it, whether the app intended it to happen or not.

In the back of my mind, I sometimes wonder how something like Emacs could be made with modern browser canvas, to enable cheap rich multimedia, while retaining the ability for inspection and user-programmability. Introducing arbitrary GUIs is hard, because next thing you know, half of the stuff is drawing to canvas directly and it's all sandboxed away from you.

I think this is why it reminded me of TCL, specifically the "expect" command that can script apps that know nothing about it. From the Wikipedia page, the TCL Expect extension: "automates interactions with programs that expose a text terminal interface".

So how I imagine this "Emacs as an OS" paradigm you're describing, is that it mediates interactions with any and all apps that expose a text input/edit interface, to allow programmatic customizations.

Like I'd love to script my own shortcuts for Firefox (or other apps) - possibly with multiple steps, taking input from some config file, or sending a link to another app.. Or, as you mentioned, Emmet-style expansions that work in any input field or textarea..

If you want larger clumps of code then you have various options such as skeleton mode, but again that's something the user has to ensure happens - again they remain in control.

> But then again I wouldn't trust Apple or Microsoft to do it right

Oh hell yes!

For the sake of completeness, you can always make it automatic. All it takes is to add a function to post-self-insert-hook, and make it e.g. call dabbrev-expand if you pressed space twice. So you can have it any way you like - manual, automatic, semi-automatic. You're in full control.

> If you want larger clumps of code then you have various options such as skeleton mode

Yes. I currently use yasnippets for code. Still, my favourite yasnippet is one I use in comments - it expands "todo" into: "TODO: | -- [my name], 2019-10-29.", and similarly for "note", "hack" and "fixme". | is where the caret ends after expansion.

That's the kind of flexibility I wish my OS had. Unfortunately, it goes against the commercial interest of mainstream OS providers.

Actual example, was working with vis studio with another guy. Open a bracket and VS automatically added a closing bracket. That is fucking annoying and saves you a whole keystroke while breaking muscle memory and interfereing with our work. We had trouble turning that off.

> I use those perhaps once a week or less. If you use it say 3X a day, you'd save little time, perhaps a couple of minutes.

Maybe you only use them so rarely because they're not convenient to use.

For people learning, perhaps it could be a good thing.

I'd love to help with the CJK support in Firefox!

We see this in every corner of Firefox: the text editing, the toolbar, the context menus, the form controls, etc. Yet Firefox seems to be all about writing everything from scratch. There are many bugs filed for using native functionality, like [1], that have been open for decades with no activity. 20 years ago, it wasn't done because "it's a time thing", and since then it's racked up other bugs as dependencies because stuff's just broken.

This isn't a situation where you can tweak a couple little problems and call it done. This is a fundamental change in the Firefox architecture. Asking for more bug reports is not going to help.

[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=34572

Firefox can't just use native text edit controls. First, Firefox needs to support contenteditable, which doesn't map to an OS-supplied text box. Second, the multiprocess architecture leads to a situation where the UI process talks to the native IME API but the Web content process hosts the text being edited, so native text boxes don't work even for things that look similar to OS-supplied text boxes.

Living in Japan and using Linux almost all the time, I never remember having any problem with typing in Japanese whatsoever.

Kanji (not to be confused with Hiragana or Katakana) are different because the characters are already composed.

[0] https://news.ycombinator.com/item?id=21390039

With pinyin input it's the same in Chinese. You enter the Latin characters and the IME gives you options to select from. Also even abstracting from the single syllables you often can narrow down the selection of compound words in Chinese IME if you continue typing. So again state is important.

On a laptop, my experience is that romaji -> hiragana is as-you-type due to being unambiguous, while the default trigger for hiragana -> kanji is the spacebar, same as described for Korean. Hence my confusion as to how it's different - the individual characters certainly are, but it sounds functionally identical.

On my phone just now, typing this comment, the experience was switching to a Japanese keyboard and inputting the hiragana directly, then the kanji suggestions appeared where autocorrect suggestions normally would.

(If there had been the kind of rendering technology that is used for Indic text today back when Korean text processing on computers started, chances are that the syllable grouping would be handled as rendering-time shaping and not as an input-time IME issue.)

Additionally, Korean IMEs have a feature to convert a word into Hanja, but it's something you need to take action to invoke as opposed to Japanese IMEs offering to convert to Kanji by default.

...that's the opposite of the comments that triggered my question. Multiple people said space is needed to trigger it, then backspace to remove an erroneously-added space character.

Is the answer that they're using it wrong, and are actually inputting a space directly because the IME already acted for them?

Getting Japanese writing to work was a pain in the ass and still doesn't work everywhere.

When I switched to Linux everything just works out of the box: I can copy and paste text between gvim, xterm, etc. with no issues. I admit that this is likely due to app writers, not underlying OS. And my experience is only with single-byte characters. Just my 2c.

It's pretty popular with japanese novels/manga and traditional chinese but really hard to do without bugs on the web.

What specific IME problems do you have with Firefox on Mac?

> On Linux, it’s terrifying; I’ve never seen any app that allows input systems to work naturally, and after a week of use you get used to pressing space & backspace after finishing every Hangul word.

Do you mean you have to press space twice and erase the second space? With IBus?

Pressing space confirms the current selection in the Japanese IME, which is expected behavior. Where some Linux implementations get it wrong is they also insert a space after the word, meaning the user has to select the desired word in the IME with the space bar and then remove the erroneous inserted space.

Edit: Correction based on feedback below. Previously stated that Hangul does not have spaces.

https://www.omniglot.com/writing/korean.htm

Wrong, there are spaces between words in Korean. It’s in Japanese and Chinese that there isn’t. And in Vietnamese there are spaces between everyone syllables, even in words.

Not sure your comment on Vietnamese is accurate though. I work in a company with ~35% native Vietnamese speakers and I’ve seen plenty of multi-syllable words.

Are you talking about traditional Vietnamese (when it still used Chinese characters) or modern Vietnamese (post-French-colonialism) which uses the Latin alphabet with accents?

I never noticed (even while studying the basics myself) that syllables are space-separated.

I always saw the words (e.g. “thanh pho” = city - don’t have the keyboard on my phone) as independent units. Didn’t even recognize the spaces.

Amazing how something can be right in front of you without noticing it.

So much makes sense now. Thanks again.

The standard Vietnamese keyboard layout works without an IME layer. However, apparently most people who write Vietnamese _prefer_ to use an IME.

Also like dfcowell said, Vietnamese used to be written with han & chu nôm characters (respectively Chinese characters and Chinese-like characters created by Vietnamese), a lot of which are encoded in Unicode. Hence the existence of the CJKV acronym.

From Unicode spec:

> Although the term “CJK”—Chinese, Japanese, and Korean is used throughout this text to describe the languages that currently use Han ideographic characters, it should be noted that earlier Vietnamese writing systems were based on Han ideographs. Consequently, the term “CJKV” would be more accurate in a historical sense. Han ideographs are still used for historical, religious, and pedagogical purposes in Vietnam.

Are you using ibus? I'm on debian and once a character is finished, it automatically moves onto the next character (unless you press space, enter, etc). All I have to do is ctrl+space to switch input methods.

I've had issues with terminals like the character building not working in alacritty. The only major annoyance I've found is having to install and configure ibus/ibus-daemon, and CJK fonts.

Right, I prefer slim systems and I typically uninstall everything input method related that my distro has chosen to preinstall. I cannot read or memorize a single CJK character, so why would I need that.

Aditionally programming and computers for me means English, although that is not my mother tongue. I would never install anything in my mother tongue.

Basically I use my mother tongue (and a couple of other European languages I speak) only in Email, chat or maybe some web form. I can feel your pain though, because 10+ years ago we had the same problem with the couple of non-ASCII characters you need in most European languages.

In order to have the situation in Linux improve there just need to be enough CJK contributors to fix existing bugs. And reviewers / unit test cases to make sure we Westeners don't break it again with our next commit.

Yes, I'm exactly talking about this mindset. This is basically why Linux has such poor input method support. Because English has a special privilege of not needing input methods to be input in, combined with the fact that the majority of Linux application programmers use English only, that means basically all apps that don't consider i18n seriously are by default 'wrong', opposed to apps running on Windows/macOS which are by default 'right'.

> Basically I use my mother tongue (and a couple of other European languages I speak) only in Email, chat or maybe some web form.

Does that mean European languages are able to being input without special input methods?

> I can feel your pain though, because 10+ years ago we had the same problem with the couple of non-ASCII characters you need in most European languages.

The non-ASCII characters fit in the character array model that most western people think in, and as a plus they are fittable in the upper half of ASCII.

Asian CJK languages require a different model from the western ones.

> In order to have the situation in Linux improve there just need to be enough CJK contributors to fix existing bugs.

It's a failing fight. That only works on an ideal world where every program has enough contributors. Thats not true.

>Yes, I'm exactly talking about this mindset. This is basically why Linux has such poor input method support.

Don't understand me wrong. That input methods are useless for me, because I know zero CJK characters, does not mean I think they are useless to everybody or Linux in general. How would I help the CJK users by having something installed I never use and I have no knowledge to use?

Yeah, but every app would 'just work' if we have a level of indirection with an IME by default, even for languages with Latin characters.

I mean, there is a reason why Windows & macOS all selects a similar architecture on text inputting.

Yes, there is a reason. Microsoft and Apple want to make money in CJK countries. And they have architects that make system-wide decisions.

That is not how Linux works. Companies contribute where their business is. That is server or embedded. After Canonical closed bug number 1 https://bugs.launchpad.net/ubuntu/+bug/1 no big player is interested on Linux on the desktop anymore. Individual contribute what they are interested in and what they know best. I fear most Westerners don't understand the challenges of CJK and other more "complicated" scripts. I myself "blame" Americans if they do it wrong and something accepts only 7 bit ASCII. I can fully understand if CJK or right to left people blame us "8 bit people" (character set, not coding) for doing it wrong. We just don't get it, that's a fact. But I don't think studying Korean etc. is a realistic solution. The only way to change it is to have more people and companies contribute that a) need it and b) really understand the user needs.

I cannot talk about East-European languages or really small languages. But for the bigger West and Central-European languages the answer is yes.

Every character is either on the national keyboard or (if typing another language) can be typed using dead-key accent or AltGr. Sometimes the compose character is needed, but I need that so rarely that I forget the combinations...

Yes. Dead keys are logically like a tiny IME, but on Windows and Linux they aren't API-wise IME-related.

Linux isn't just for programming. I personally hate that I have to dual boot because ubuntu can't do what windows or mac can. Anyhow, English has plenty of words containing non-keyboard characters, they're just infrequent.

[1] https://stackoverflow.com/questions/6841333/why-is-subtracti...

More like we stand on piles of broken shit. A lot of "hard" things in programming have very little fundamental complexity. They are hard because at some point in time someone made a compromise that no longer is relevant or makes any sense. It is possible to redesign those things to be simple, but modern developers are conditioned to fight this at every step. The idea that the first thing you should try to do when faced with extraordinary complexity is to bypass it is completely alien to people whose daily jobs consist of continuously banging their heads against convoluted frameworks and deficient APIs.

Time handling is a great example. Until you get into general relativity, time itself is extremely simple and uniform. But human representations of time are extremely convoluted, because historically they are based on astronomy. Here is the kicker: 99.99% of all software has nothing to do with astronomy, and yet people chose to inject those complexities in core structures that track time. (For example, Unix timestamps have leap seconds, which is a source of innumerable bugs and ridiculous convolutions in code to prevent those bugs.)

If you don't want software to interact with humans, or you only care about duration rather than human datetime, then go ahead and use UT1. However, as long as your interacting with people, you're going to need to fix people first. You should start with these primers if you're going to fix how humans interact with time:

- https://qntm.org/abolish - https://qntm.org/continuous

For instance use straight timestamps where there is no need for calendar manipulations.

I am sympathic to this argument and I’d also totally join a movement to ignore DST or uncouple years from earth rotation

Just a little niggle, but I think you meant to say special relativity; special relativity is about the relationship between space and time, whereas general relativity (I believe) is more about gravitation and other things that I definitely don't understand.

Time zones are the worst because they are political tools, changed for political reasons, often with very little lead time. I think of them as a one way function mapping universal time onto the local political reality.

That is, "convert time into time in seconds since the epoch" is the hard part. All you've done is make it someone else's problem, while the question Skeet answered asked specifically about the time difference between 1927-12-31 23:54:08 and 1927-12-31 23:54:09 in Singapore.

The POSIX spec exists for this reason. This feels like a variation of "omg utf-8 is hard" which is why people embed utf handling in the language python2 -> python3

What does POSIX have to do with anything? Eg, the tz database isn't POSIX.

I'm sorry if I came across as churlish. Do you feel this is a problem of concept in abstract, or in java, or in java implementation or what? I feel it's a problem which I solve, by standing on the shoulders of smarter people and using the code they wrote. I don't reimplement the wheel because I distrust my own wheelmaking skills.

I'm sure there are heaps of non-trivial corner cases I don't have to deal with and I do not mean to minimise work done to explain a problem.

When it comes to timestamps, how do you handle midnight for example? 24:00 today represents the same instant as 00:00 tomorrow. The difference might matter to the user but disappears when converted to Unix timestamps.

Your response took things for granted.

dvt originally wrote "We stand on the shoulders of giants."

You commented "I feel it's a problem which I solve, by standing on the shoulders of smarter people and using the code they wrote."

It came across as totally missing dvt's point.

I picked Russian because I was thinking of an essay by Douglas Hofstadter. Quoting myself at https://news.ycombinator.com/item?id=16289783 :

> I think it was in Godel, Escher, Bach where he talks about the problem of translating a line from Russian, along the line of "He lived on B____ street." It's possible from the story, which takes place in a real city, to figure out which street that was. Let's say it was "Main Street". Does the English translation keep the original Russian word and initial? Does it translate to "Main Street" and replace the "B" with an "M"?

You can use a Russian-English translator to get a translation into English. But that doesn't help understand the hard problems involved, which was the point of Hofstadter's essay. (See, for example, his "The Shallowness of Google Translate" in The Atlantic; HN comments at https://news.ycombinator.com/item?id=16267363 and in dupes.)

You should still be careful how you define a _second_. A POSIX second is not the same as an SI second. A POSIX second corresponds to 1/(24 * 60 * 60) of a day. An SI second is defined using atomic time measurement. It means that the duration of a POSIX second depends on the length of the day, and some days are longer than others due to leap seconds.

The worst part is that most tools rely on POSIX timestamps instead of TAI timestamps (based on the SI definition). One of the consequences of using POSIX timestamps is that taking the difference between two timestamp does not safely return the number of elapsed seconds.

I don't think that is quite accurate. Traditionally Unix Time seconds are SI seconds, that tick in sync with UTC and atomic clocks (modulo clock error). To account for leap seconds, Unix Time has traditionally been discontinuous when a leap second occurs. Each second that ticks is still an SI second, but the sequence of Unix Time around a leap second will be X-2, X-1, X, X, X+1, X+2, etc.

This discontinuity means that every Unix Time day has exactly 86,400 seconds cumulatively, even though some days tick 86,401 elapsed SI seconds. So it is true that computing a difference between two Unix Time values does not give you a reliable measure of how many SI seconds have elapsed, as you say. But the individual seconds are still all SI seconds.

More recently it has become popular to smear the leap second instead. Google pioneered this approach in 2008 (https://googleblog.blogspot.com/2011/09/time-technology-and-...) and now advocates making it standard (https://developers.google.com/time/smear). This approach does vary the actual length of a second during the 24 hours surrounding a leap second. But this doesn't come from POSIX, it's a departure from previous practice that is designed to avoid a discontinuity.

> How any changes to the value of seconds since the Epoch are made to align to a desired relationship with the current actual time is implementation-defined. As represented in seconds since the Epoch, each and every day shall be accounted for by exactly 86400 seconds.

Another fact that I find interesting is that the use of discontinuities or time-smearing of the leap days are actually modern approaches. Nowadays, POSIX and TAI differ by an integer amount of seconds on regular days, and a fractional amount when smearing the leap-second day. At the beginning, the leap second duration between POSIX and TAI was smeared over years from one leap second to the other. So the difference between both timestamps would change continually other these periods!

My main takeaway is that POSIX timestamps are based on calendar days while TAI timestamps are based on physics constants. POSIX timestamps are a tradeoff: they're not suitable for time computations (durations, or even uniquely identifying an instant when discontinuities are involved) but they don't require a leap seconds lookup-table to display time for humans... in UTC... Which means you still need lookup tables for timezones, countries, DST, calendar changes, etc. Basically my opinion is that POSIX time is neither the best to precisely keep track of time nor to format it for humans. It just sits in between these two use cases because it was good enough at the time and then too hard to change.

[0]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...

Do you have any reference to a system that was implemented this way? I have never heard of such a thing and I don't see how it could be implemented.

The insertion of leap seconds is unpredictable and occurs when the International Earth Rotation and Reference Systems Service determines that the difference between UTC and UT1 are approaching 0.9 seconds. Leap seconds are generally only decided six months or so before they occur. So you don't know at the beginning of the year how many leap seconds the year will have. So it doesn't seem possible to smear over a year.

> UTC is a discontinuous time scale. From its beginning in 1961 through December 1971 the adjustments were made regularly in fractional leap seconds so that UTC approximated UT2. Afterwards these adjustments were made only in whole seconds to approximate UT1. This was a compromise arrangement in order to enable a publicly broadcast time scale; the post-1971 more linear transformation of the BIH's atomic time meant that the time scale would be more stable and easier to synchronize internationally.

Wikipedia sources it to a book [1].

[0]: https://en.wikipedia.org/wiki/International_Atomic_Time#Rela...

[1]: McCarthy, Dennis D.; Seidelmann, Kenneth P. (2009). Time: From Earth Rotation to Atomic Physics. Weinheim: Wiley-VCH Verlag GmbH & Co. KGaA. ISBN 978-3-527-40780-4.

See: https://infiniteundo.com/post/25326999628/falsehoods-program...

As an example: I use a Swedish keyboard. If I'm in a situation where ö is missing from the input (because the app is stuck in an en-US keyboard layout say, so I have to hit a modifier ¨+o to type an ö) then there is a very good chance my communication with my colleagues would just naturally be in english instead. I'd sigh and just give up using it. Rather than using an o for my ö I'd just type it in english instead.

Is this a thing in e.g. asia, israel, or the arab world? Do kids that speak english communicate more in english in cases where the input doesn't let them communicate easily using their preferred script? Are there new "hybrid" languages popping up in electronic communication where languages that use non-latin scrpit are written in latin in e.g. text messages? (You could argue that emoji is just that but the other way around I suppose)

So yes, even when "localized", hardware and software is usually not fully adjusted to the language. And for German, which is relatively close to English, the problems are probably relatively small. I cannot even fathom how big all these issues for more distant languages must be.

And to answer you questions: yes, these pain-points lead me to sometimes prefer English and to avoid special characters. I even avoid spaces in file names as too many programs I worked with in the past struggle with that.

Oh, and I always say German works still great. In Finnish words have many 100s of different endings (several concatenated + combinatorics explain how this is possible) and predictive inputs just don't work. Probably it could be improved with massive semantic support, I am not an expert in that field.

Switching to English is way overblown a reaction. Two Hungarians chatting in English (unless there are non-Hungarian speakers involved) seems extremely weird to me. It may be partially that English is really foreign for us, while it's pretty close linguistically to Swedish, both being Germanic.

For what it's worth, I have a Swedish friend who almost always posts in English, even when he's talking to his family and other people he knows in real life. He'll switch to Swedish occasionally (and Facebook's autotranslation is very understandable), but 90% of the time he uses English.

That wouldn't explain it, because we do the same in Dutch: just leave off the diacritics. Diacritics are pretty rare in Dutch, though.

Hungarian is also redundant enough that you can even replace all vowels with just one and still be understandable. Most of the meaning is carried by the consonants. E.g. "Szia én vagyok Péter, hogy vagy?" -> Szii, in vigyik Pitir, higy vigy. (sounds obviously wrong, but very understandable) Retaining the vowels but collapsing all consonants to one would be more destructive to the meaning.

in Tunisia they write arabic using the latin script.

for example: ahla أهلا := hello

for sounds that are inexistant in latin script they use numbers:

3 == ع , for example 3andek, عندك := you have

5 == خ for example 5amej, خامج := dirty

7 == ح , for example n7ebek , نحبك := I love you

9 == ق, for example 9rib, قريب := near

.

[1] And when the affected user base is small enough, even larger problem can take a lot to fix. I use a third-party IME that is relatively known among programmers, and early this year Google Chrome began to crash seemingly at random when you type Hangul. I tracked the root cause and I believe it is ultimately Chrome's fault, but I was too busy to file an issue at that time and the IME has adapted to Chrome to avoid triggering the crash next month. I believe Chrome still crashes when older versions of the IME are in use.

I'm aware that third-party Chinese and Japanese IMEs are a thing, but what user-facing difference does the third-party IME you mention provide in the Hangul context?

- Support for less-popular or customized keyboard layouts. From time to time popular OSes had a wrong version of supported layouts as well.

- Deterministic autocorrection. For example, normally an initial jamo (e.g. ㄱ) should be followed by a medial jamo (e.g. ㅏ) and not vice versa, but many IMEs offer a feature that automatically swaps them. With a carefully designed layout this can catch lots of transposition typos.

- Custom candidates for special characters, as Korean IMEs show special characters as candidates when a jamo is being "converted" to Hanja (popularized by MS IME).

The IME in question [1] also allows an extremely customizable input system, to the extent that it forms a soft of wholesale DSL for Hangul IMEs.

[1] http://moogi.new21.org/en/ngs/index.htm

The Arab world uses a lot more of the Latin alphabet, but they don't write in English. For example مرحبا would turn into mar7aba. Here's a wikipedia article on the phenomenon. https://en.wikipedia.org/wiki/Arabic_chat_alphabet#Levantine...

With China, the market's so big that there's an entire ecosystem of homegrown (or Chinese-modified) software and websites, everything from office suites to web browsers, and I have to imagine those support Chinese input just fine.

With Japanese, generally the same software and sites are used, but Japanese support is also pretty good across common software.

Some niche or open source software struggle with IMEs in general, but in those cases, the solution is just to not use that software.

As for mobile, you can generally find the language-specific ones when pressing and holding some character when you're on a normal qwerty keyboard.

For Scandinavian:

å: altgr + w

ä: algr + q

ö: altgr + p

æ: altgr + z

ø: altgr + l

Should be available on all OSs. I never have to switch layouts when switching between languages. Still struggling with finding an actually nice IME setup for Japanese and Chinese, though, and I've spent some time...

On Linux: setxkbmap -layout us -variant altgr-intl

Regardless, I really think input methods should be kept out of scope for all apps. There's nothing I hate more than app developers trying to "solve" it for me. There are already people who work full-time on this and users who have their systems set up as they want. There's no way an app developer is going to help more users than they break things for.

The only exception I've seen is Google's web CJK input for Translate, but they fall under the "huge amount of resources" category.

Context: I write in Swedish and Japanese on a daily basis, spent some time learning Chinese and sometimes have to retype things from other languages. Using vanilla iOS and Android on mobile, Linux on desktop. CJK has been the only struggle so far. There's enough tech-literate people with those native languages that there definitely are ways to get good configuration for any OS, for Linux you just have to figure out how, which can be tricky to find if you don't read the language.

I can imagine Arabic and Vietnamese can be worse.

YMMV, but I got used to it pretty quickly and I type Swedish all the time, way less annoying than doing the Alt-shift dance to switch layouts whenever I go from coding to chatting at least ;)

That said in Linux/X11 you could configure everthing to your likings. I must admit I prefer complaining...

Right-alting for åäö is just way less cumbersome than doing the same for @${}[]\~| or having to switch layouts when switching contexts.

In Ukraine, no, we don’t. Kids don’t generally know English enough that they would want to speak it with peers, and there are simply no reasons to do so.

“Translit” (writing Ukrainian/Russian with English characters) was a thing in SMS days, when typing in Cyrillic would shrink your SMS length in half (because that’s how SMS encoding worked, I guess), but now practically nobody uses it anymore.

Ok that's exactly the type of thing I was thinking about. Funny it was born from the limitation of 140 chars rather than from actual input difficulty though.

But back in the 90s to early 2000s, translit was pretty big on the internets due to the dozen of different Cyrillic codepages and some software supporting only 7-bit characters (specifically Fidonet exchange software and newsgroup nodes).

Just the fact that I still to this day run into mis-encoded diacritics in movie subtitles, shows how ‘US first’ ASCII caused a lot of headaches for many years.

(BTW: you probably have been told by now, but your nickname is a common slang word in Russia.)

That said, I type mostly in English anyway. My reason for using a US layout is that it is more ergonomic for bracket/brace/semicolon heavy programming languages. Braces in particular are an annoying chord with Swedish keyboard layouts. Especially on OS X, which for whatever reason uses three part chords (alt+shift+8 or 9).

Yeah, transliterating arabic to english is a thing:

> With smartphones, the relationship between informal, chatty Arabic and formal written forms has become more complicated, giving rise to a hybrid known as “Arabeezy” – or Arabic written with Latin characters and numbers to represent letters that have no English equivalent.

--

https://www.thenational.ae/arts-culture/books/duolingo-s-ara...

Or see this wikipedia article: https://en.wikipedia.org/wiki/Arabic_chat_alphabet

ä -> ae

ö -> oe

ü -> ue

At least that is what I was taught. When writing Spanish some of my friends have replaced ñ with gn as in Italian or ny as in Catalan.

As the input method of CJK languages are significantly different from English (compared to the relatively small difference), every app just allows the locale's input method.

Nobody tries to communicates with only the English keyboard, b.c that is outright impossible (while in Swedish it's possible in worst cases).

Hmm, I've never encountered situations when only English keyboard is available for communication (except for when someone is setting up a new Linux machine).

But even if there is such situation, I don't think that would happen, I'm not sure if Swedish is similar to English but English is cognitive load here. Even with people that are proficient in English, I cannot imagine myself communicating with the English language.

I remember myself resorting to online keyboards though(while setting up Linux).

Both, in my part of India. The latter used to be much more common in the feature-phone era, writing in Tanglish (Tamil words+English script), but seems to have become much less popular now. Not sure why, perhaps swype-type keyboards have made English so much easier to type.

The former certainly still happens a lot - I know a bunch of people (including myself) who would prefer to communicate in Tamil, but there's enough friction (not having a Tamil keyboard, having to get used to new keyboard layouts, the Tamil keyboard being buggy often enough, ...) to make it more appealing to go with English for normal conversation.

AFAICT, Swedish keyboard has a dedicated key for ö. You just press it. I don't understand how an app can get stuck on keyboard layout if you set the system keyboard layout to Swedish. Could you give an example?

Or are you're actually using US keyboard with Swedish layout and there is no ö key on it?

> Could you give an example?

It was a hypothetical, but in windows you have a per-app layout (so you can choose multiple keyboard layouts in windows and an icon in the systray will let you choose the active one) and that one often spuriously switches so suddenly you hit the Ö key and it produces a ';'. A better example is perhaps when I'm travelling and composing an email from someone elses computer at my office in Britain back to my office in Sweden. I'm not going to bother switching the windows settings just to write the email on that machine. Or if I'm logging on to a VM with only english layout so hitting my Ö key will produce a ; If I needed to type anything inside that VM desktop for whatever reason, I wouldn't bother trying to write Swedish even if the person I was typing to was Swedish.

> Or are you're actually using US keyboard with Swedish layout and there is no ö key on it?

Personally I actually do use a US keyboard wit ha Swedish layout, which works fine. If I hit the key labelled ';' it will show an Ö on screen. The reason is I wanted a specific type of keyboard that didn't exist with Swedish layout, and the layouts are almost equal. It's just one missing key between lshift and Z that doesn't exist in US layout.

Honestly, English is uniquely suited to computer input. A narrowly-defined charset maps well to a situation where you can only fit so many keys on a keyboard or on a screen. I also sympathize with difficulties implementing languages that have ligatures, right-to-left reading, or other significant differences. Non-discrete chars just don't map well to a world of ones and zeroes, because representing all is the simplest option but has O(n!) complexity in such languages (assuming you can combine all, which is probably not possible, but you get the point). I have a great deal of sympathy for maintaining such complexity for what can be a very small part of your user-base (that is can communicate in only one foreign language).

I personally have muscle-memory for my own programming-optimised adaptation of Type 2 Dvorak, regular German layout, and UK/US/International variants of English keyboards, each in Mac and PC variants. And, I suppose, in mobile phone and tablet screen-keyboard variants.

It always takes a few seconds to adapt to whatever I’m sitting in front of, and I have a lot of practice with this compared to the average person.

At least in German, there are standard substitutions for when umlauts and sharp S aren’t available for technical reasons:

ä -> ae

ö -> oe

ü -> ue

ß -> ss (historically “sz“, which you still occasionally see, particularly where there is Hungarian influence; Swiss German tends to forego ß altogether)

Multilanguage support is a function of how many customers demand it, and aside from English and Japanese, there hasn't been enough economic incentive to take it seriously.

For the record, it's a habit you get used to rather fast; nowadays I use the ¨ modifier for ä and ö even when I'm using a Finnish keyboard.

How often does this happen? I use Ubuntu and it seems like every application respects the OS's keyboard settings.

In addition to what's mentioned in those articles, on larger documents rendering speed also matters; word cache (in addition to mentioned glyph cache) helps somewhat, but complicates other things even further, and calculating total document height (for scrolling) still requires to calculate all the positions, wrap the lines, etc, which in turn requires to render all the text first. Perhaps one can introduce yet another mechanism to only estimate that at first, and then elaborate the estimation in background, but that's yet another opportunity for bugs to creep in (and additional complication, of course). On the contrast, the programs that just use character grids (and perhaps don't handle bidirectional texts well) can be both faster and simpler, while still are quite capable of rendering texts, even with various styles/decorations and occasional images, in many languages.

There seems to be plenty of accidental complexity even in handwriting, but once it is used for computing and text manipulations are involved, the task appears to be much harder than it has to be.

Yeah... and that's why FireFox & Eclipse still doesn't get CJK character input right. The percentage of people that must use characters not included in ASCII is much bigger than people who do. It is worth the trouble, please consider users outside of the US & England.

Imagine a world where everybody speaks at least one common tongue, we could have so much more peace and understanding.

English becoming the esperanto of the world is an accident but I don't care which language it is, as long as it has less than 255 letters. (Hawaiian would have fit into a nibble which is cool)

Why not Chinese? It would have many advantages:

- Text layout would be much easier (Chinese fonts are mostly mono-space).

- In addition to the first point, Chinese characters can be easily used for vertical text. Looks much nicer in narrow, tall buttons than latin letters.

- Chinese characters are (mostly) pronunciation independent. No they/there/their anymore.

- It has been shown that dyslexic children have less problems recognizing Chinese characters than latin letters.

- Chinese characters are part of the culture heritage of billions of people (both Chinas, Japan, Korea) and not just something they learn at the university.

- Old-school C enthusiasts can now have meaningful variables names while keeping their self-imposed three-character limit (or was it four?)

- Finally, everybody will use wchar_t

- Edit: But there is one disadvantage. There is no Chinese character for "/s" which some people here seem to need

Like I said, Hawaiian would be cool since it fits inside a nibble with 16 characters, although the words tend to get long.

As a software engineer and cousin of Stitch, Hawaiian gets my vote! :handclap:

Hmm... it might not be well known to the western world, but AFAIK most parts of the world learns English as their second language. At least East Asia learns English pervasively throughout your life. We start learning English when we're five.

But still, English is not the only language, and we want to communicate with other people. Just that people can use English doesn't mean they prefer English & Latin characters to communicate. You can't just er... force people use Latin characters for the sake of programmers' comfort. Will you use your terminal if the terminal demands you to press space & backspace everytime when you're typing in English characters?

This is more than programmer convenience, this is about system simplicity, absolute correctness (too many bugs because of unicode and complex layouting) and above all about _forcing_ people to use said lingua franca to become comfortable speaking it.

If you want your own writing system, fine, but put the effort in and build it yourself apart from the common bug free code base that the world shares, it should be less buggy overall anyways, than a system which tries to speak every language and support every writing system.

The reason the writing system in computers are simple is because the latin characters are simple, and the majority of early computer users (which the majority of was English speakers) didn't feel the need of sophisticated systems.

I (often) find that complexity that is inside the western culture is warranted, while complexity not in western culture is not.

While not a great analogy, think of TTS. English TTS is (at least from what I have heard) not that easy (encodable in logic, but not as simple as mapping characters to audio). That resulted in some sophisticated TTS systems, where it can handle lots of different phonetic structures.

Compare this to Hangul (the Korean character system) where every character is composed of several mini-characters which have a unique mapping to audio. Basically for a minimal viable product you can just convert text to decomposed format(NFD) and map the char points to audio.

Now, let's say that some company (an arbitrary choice, Facebook) decided to make a global TTS product, first in Korea. They just decided to map the char points to audio and post-process the audio. This TTS architecture obviously can't handle English without some major architecture changes. Then Facebook decides, this is a problem in English where the phonetic structure is unnecessarily complex, and won't provide TTS service to English users.

This is something similar to what CJK users face, that people just won't make products that work reliably with CJK.

The thing with Korean is that "nobody" speaks it. Had early computers been developed in Korea and would everybody speak Korean I would argue now that everybody should just learn Korean. But tough luck, it didn't. Our best shot at a lingua franca and simple computer systems(simple in the sense; as little complexity as possible) is with english.

For starters, while young people may feel more comfortable with occasional English older people don't, and your proposal will make them much more uneasy. Smartphone penetration rate around the world [1] can easily reach 50% for many countries and that would include many people unable to understand English---I live in Korea and older people are technologically no inferior to younger people in that regard. Impossible when you force them to use English.

[1] https://en.wikipedia.org/wiki/List_of_countries_by_smartphon...

You guys truly are wonderful unicorns. How do you do it? (serious question in that regard)

On topic, I think the ultimate solution is to have whatever language input/output be essentially a placeholder, and then some general "switching" system which puts whatever language a user prefers. It should be able to switch on-the-fly totally independently of application state. (same thing for numbers, outputting decimal or hexadecimal should happen on-the-fly, same model value but different user view)

Commercial OS's tend to be pretty invasive these days, but the one thing where we need more of them, and better tooling, is definitely text i/o.

> Hangul would also be a nice lingua franca writing system, too bad its very local

By the way, Hangul was tailor made for Korean with its relatively simple CGVC syllable structure. I doubt it can be generalized much---Hangul is important because it is one of the first featural alphabets, not because it is a universal (or at least adaptable) featural alphabet where it isn't.

[1] English has about 2 billion speakers (400M native + 750M L2 + 700M foreign), so you need another billion speakers or two to make your proposal real.

[EDIT] was "CJK", but I'm not sure about Chinese and Japanese.

A language is a better candidate for a lingua franca if its spoken everywhere a little, than somewhere a lot, e.g. Mandarin.

And this is not about cost, but about Bugs in critical infrastructure. I'd get rid of smartphones and unicode in a heartbeat if it meant bug free command line applications, no malware, and medical and infrastructure that doesn't crash.

I think its great when non english workplaces require english. It gives people an incentive, because humans are a lazy species. And once you learned it you can use it on your next vacation, your next online encounter, and who knows where.

Having computers only do english would have given a similar incentive, and I'm sad we didn't use that opportunity.

English is a germanic language. You say "gut" instead of "good", "zu" instead of "to", "wir" instead of "we", "Wetter" instead of "Weather", "nichts zu tun" instead of "nothing to do".

You will not be doing as much effort as someone from a totally different native language. The CIA calculated that it requires over 4 times the effort to learn Chinese for an English speaker than learning French. It also applies back, a Chinese will find it very hard to learn proper English.

Learning English for a german student is as easy as taking the train and spending some days in England.

Now for most Chinese, or even Indian(they have more than 20 languages) people, or Arabic,it is not so easy.

The fact is that they are in a disadvantage,not in a leveled field and they don't like it.

They are the majority of the population in the world, so if they develop their economies they will force everybody else to learn their native language, not the other way around.

And of course I'm at an advantage, the very fact that I'm german would put me at an advantage even if english was as far away as possible from german, because I have the resources, I have free education and I have free medicare, plus 400 years of the subjugation of the third world in my back. So what. Life isn't fair, the worlds top 30 People own as much as the bottom 3 Billion, thats unfair.

Instead of bickering about which language would be fairest to adopt you should help make everyone adopt SOME language so that they can organize and exchange. So that they can understand for themselves what other are saying instead of having to rely on potentially biased news. So that they can talk to each other, for sympathy and empathy and understanding.

My bet is on English for that one.

Plus it seems that it's not so hard after all:

http://www.hole-in-the-wall.com/Beginnings.html

At the bottomline I find it amazing that a human managed to draw symbols in columns or rows in any direction for centuries, but computers still have trouble with it. Maybe we should blame computer makers, not languages?

As for writing direction: Simple tools allow for complex creation. Complex tools allow you to only create simple things.

And boy are computers complex.

That seems a little extreme to me. It seems like it might have been better to just adapt writing systems for computer usage. Originally a lot of computer systems didn't have lower case letters, japanese tended to just use katakana, etc. Sure, it kinda forces everyone into an alphabet or syllabary, but language adapting to medium is hardly a new thing.

What they all lack is reach, and thats the most important feature when it comes to a common tongue. (That and having a phonetical system ^^')

  · Break the text into runs.
  · Calculate BiDi at paragraph level.
  · Shape every run to get its length and possible line breaking points.
  · Arrange the runs (now only have length, height, text isnide it and some breaking properties — glyphs and glyph positions are dropped) into lines.

In most times (like appending text) you do not to do much recalculation.

Rendering speed hasn't mattered since years ago. Browsers can render megabytes of pure text (hours of reading material) in seconds. LaTeX can typeset hundreds of pages in seconds.

I do, that's one of the reasons I'm trying to handle Unicode and proportional fonts in the first place. Given the circumstances, it is indeed preferable to do so, but given that it's basically encoding of approximated sounds (well, potentially information in general) on a 2D plane, it is a rather complicated (both code-wise and computations-wise) way to achieve such a task.

> Rendering speed hasn't mattered since years ago. Browsers can render megabytes of pure text (hours of reading material) in seconds.

Chrome only started using "complex path" for English texts a few years ago, employing mentioned word cache. Rendering megabytes of text in seconds is indeed achievable (especially if the words are cached, and it's not megabytes of integer sequences), but still complicates the rendering, and still slower than to open and navigate through the same megabytes of text in, say, less or vim, or even links -g. It's not an issue if you don't have to do it (or to use a program where it wasn't optimized well), but if you do, it is.

Edit: added a reply to the second part.

I don't know the official name or who came up with it, but I use Slack's entry format exclusively in every application.

:thumbsup: :pelipper_blushing: :angry_cat: :apple:

If the application can detect that as an emoji and swap it out, fine. If it can't, I don't change my format. My preference would be if applications left emoji in that format, and just rendered them differently at display time.

The advantage of having emoji just be a purely clientside rendering feature, and behind the scenes all fall back to normal text is:

a) they can be easily aliased across multiple languages (:cat: :gato:)

b) if you paste an emoji into an application that doesn't support them, you don't get an unrecognizable character. Progressive enhancement!

c) it's accessible when copied and pasted into a pure UTF-8 text format. It's just better blind-accessible in general.

d) it's more forward compatible. I can use :cthulhu: right now without waiting for it to get added to the standard.

e) get rid of modifiers. Like, seriously, just get rid of them. Emoji aren't programmatically generated, you still need to draw one image for each modifier combination, and you still need to program support for each one. So, what's the advantage of using modifiers over just adding multiple glyphs? They're just there to save space in the character list, which is only a problem because emoji are in Unicode. :smile: :fake_smile:

f) better support for custom emoji in general. Basically every platform has custom emoji, and it's weird because half of your emoji are standardized and half aren't. And then whenever new emoji get added to the standard, if they conflict with your custom emoji your app breaks.

I would hesitate to standardize emoji at all, beyond having a consortium that says, "this is what :thumbsup: means, you can extend on top of this as you see fit."

It feels like extra complexity for no benefit other than, "we need a standard".

Modifiers and combinators aren't exclusive to emojis, but apply to all kinds of glyphs in other languages and writing systems as well. Arabic script even has some common ligatures for common expressions.

A lot of complexity simply doesn't stem from emoji in Unicode, a lot of the complexity comes from all the writing systems that Unicode supports. Admittedly, emoji are kind of an oddball addition to Unicode, but they're by far not the most complex part of it.

If either of these were common (e.g. slack bots that tried to parse semantic meaning from regular text rather than responding to commands; or Slack logs of OSS communities being public-access on the web) then you’d see a lot of people up-in-arms around the fact that these custom codels are used.

But since text in these group-chat systems is private, ephemeral, and mostly a closed garden, it never bubbles up into becoming an issue anyone else has to deal with.

(Though, on a personal note, I wrote my own VoIP-SMSC-to-Slack forwarder because Slack is a much better SMS client than any of the ones built into VoIP softphone apps, and I’m irritated every day that Slack auto-translates even Unicode-codepoint-encoded emoji from a source postMessage call, into its own codels in the canonical message stream. I don’t want to send my SMS contact “:thumbs_up:”, I want to send them U+1F44D!)

Let's say Matrix supports custom emoji, and it wants to interoperate with Discord.

Does representing those user-added emoji as custom Unicode entries instead of plain text make it easier or harder to interop with Discord?

One way to encode a ‘custom emoji’ would be encoding it as a variation of some existing emoji. Use an as-yet-unused variation-selector on top of an existing emoji codepoint, and then “render” that codepoint-sequence on receipt by the client back to an image (but in a way where, if you copy-and-paste, you get the codepoint-sequence, not the image. In HTML, you’d use a <span> with inline-block styling, a background-image, and invisible content.) This is pretty much what Slack was doing with the flesh-tone variation-selectors, before Unicode standardized those. But you can do it for more than just “sub”-emoji of a “parent” emoji; you can do it to create “relatives” of an emoji too, as long as it’d be semantically fine in context to potentially discard the variation selector and just render the base emoji.

Or, if your emoji could be described as a graphical (or more graphical) depiction of an existing character codepoint, you could just use the “as an emoji” variation-selector on that codepoint.

Or, rather than a variation-selector, if you have a whole range of “things to combine with” (i.e. the possibilities are N^2), you could come up with your own private emoji combining character for use with existing base characters. The “cat grinning” emoji U+1F639 could totally have been (IMHO should have been) just a novel “face on a cat head” combining-character codepoint, tacked onto the regular “face grinning” emoji codepoint. Then you could have one such combining-character for any “head” you like! (And this would also have finally allowed clients to explicitly encode “face floating in the void” emoji vs. “face on a solid featureless sphere” emoji, where currently OSes decide this feature arbitrarily based on the design language of their emoji font.)

And, I guess, if all else fails, you could do what Unicode did for flags (ahem, “region selectors”), and reserve some private-use space for an alphabet of combiner-characters to spell out your emoji in. That way, it’s at least clear to the program, at the text-parsing level, that all those codepoints make up one semantic glyph, and that they are “some kinda emoji.” Custom-emoji-aware programs (like your own client) could look up which one in a table of some kind; while unaware programs would just render a single unknown-glyph glyph.

I don’t suggest this approach, though—and there’s a reason the Unicode standards body hasn’t already added it: it’d be much better to just take your set of emoji that you’re about to have millions of people using (and thus millions of archivable text documents containing!) and just send them to the Unicode standards body for inclusion as codepoints. Reserving emoji codepoints is very quick, because the Unicode folks know that the alternative is vendors getting impatient and doing their own proprietary thing. Sure, OSes won’t catch up and add your codepoint to their emoji fonts for a while—but the goal isn’t to have a default rendering for that character, the point is to encode your emoji using the “correct” codepoint, such that text-renderers 100 years from now will be able to know what it was.

So, please, just get your novel emoji registered, then polyfill your client renderer to display them until OSes catch up. Ensure your glyph is getting sent over the network, and copy-pasted into other apps, as the new Unicode codepoint. Those documents will be correct, even if the character doesn’t render as anything right now; if the OS manufacturers think the character is common (i.e. if it ever gets used in text on the web or in mobile chat apps), they’ll provide a glyph for it soon enough. And, even if the OS makers never bother, and you’re stuck polyfilling those codepoints forever, there’ll still be entries in the Unicode standard describing the registered codepoints, for any future Internet archaeologists trying to figure out what the heck the texts in your app were trying to communicate, and for any future engineers trying to build a compatible renderer. (Consider what Pidgin’s developers went through to render ICQ/AIM emoji codels. You don’t want to put engineers through that.)

Yeah I'm not gonna register my company icon.

Yes, it's not that emoji are doing anything odd compared to lots of real world languages, it's that emoji are just latin script writers' "first"/"only"/"most likely" interaction with that sort of stuff. The fascinating bit that that if it weren't for emoji a lot of these problems would still go unfixed in a lot of real languages, but because emoji are fun and everyone wants to use them we've seen a lot of Unicode fixes brought about by emoji that's a rising tide to lift other Unicode boats.

The problem is that emoji are part of a major text-encoding format, so unicode needs to adopt those. Once those were added, everyone and their mother suddenly wanted their emoji in unicode as well.

> It feels like extra complexity for no benefit other than, "we need a standard".

That sums up the whole standard for better and for worse.

[1] https://en.wikipedia.org/wiki/Unicode#Mapping_to_legacy_char...

That’s exactly the benefit. Think of Unicode as “Archive.org for the semantics of text codels.” Every time someone invents a text codel (like those examples you gave, where Slack invented their own text codels), Unicode takes the semantics behind that codel and standardizes their own codepoint equivalent to it, so that Unicode documents will have a way of encoding that text codel at the Unicode-text level.

If Unicode doesn’t do this, then people have to use other encodings on top of Unicode to specify their text; they come up with incompatible encodings of the same semantic characters; and suddenly we’re back to having to create code-pages to specify what set of incompatible codels each text stream is using. It also means that we go back to having to create OS-specific, or GUI-toolkit-specific rendering encodings to translate those code-pages into a text-layout-system specific “normalized” encoding; and thus, we also go back to having OS/GUI-toolkit specific fonts.

> Emoji aren’t programmatically generated.

No, not usually, but they can totally be programmatically consumed. They’re machine-readable! The modifiers allow emoji to be “structured text” in the ML sense. Since there’s one codepoint that always means “sad face”, an algorithm can attach a meaning to that codepoint apart from its modifiers, to do e.g. sentiment analysis. It’s much harder to learn when you have 100 different “sad face” codepoints; let alone when different documents use different incompatible encodings with different codels to refer to the same “sad face.”

> better support for custom emoji

That’d be like saying a dictionary should better support private in-jokes. Unicode, like a dictionary, watches for when things become common or important to define, and then defines them. Until then, it considers them “someone’s attempt at injecting a gibberish in-joke into language.” In-jokes don’t need an entry; but as soon as an in-joke becomes a word, it does. Because people don’t look up in-jokes, but they do look up slang (= ascended in-jokes), so if you want your dictionary to be useful, you’d better give them a definition for slang terms.

Actually, that’s a very good equivalence: emoji are to Unicode as slang terms are to dictionaries. In both cases, people think it’s ridiculous that the authors of the text would include them; but in both cases, the usefulness of the work would be hampered if they didn’t.

Please, please, PLEASE don't do that.

Have you ever tried pasting any kind of code (especially C++) into Skype? It always turns into a fiasco of stupid emojis. And even if you could disable it on your client side (which you can't even do), it would turn up as emoji garbage on the receiving end - and you might not even know/realize!

Just thinking about "let's show everything that's might be an emoji as an emoji" makes me shudder...

I'd also note that "you can alias it for different languages" is bound not to work. The same word means different things in different languages - which ones do you accept as aliases? How are native speakers supposed to know when their word for something can't be used because it means something else in any other language? I mean, look up the German adjective that means "fat"...

The major issue I find with it is there aren't limits to how many are added. Emoji are growing at a fantastical pace, much like a tumor, because "if X got added, why didn't Y?", and so on.

As the other child reply mentioned, this is latin-centric. Totally useless for people that can't or don't want to speak English, and having to duplicate it for every language defeats the entire purpose.

Does it?

If you want accessible emoji, you need a display label for each emoji in every language that the client supports. Whether or not the emoji is represented in Unicode doesn't change that.

Also when users are entering emoji into a client application, they need a way to quickly filter and get to the emoji that they want -- that requires having a label in their native language, and putting emoji in Unicode doesn't fix that either.

In any practical setting, accessibility/input means you need multiple labels for different languages anyway. So why are we trying so hard to avoid them in the final text representation? If :cat: :gato: :貓: were part of the emoji definition in a big list somewhere, it would only make it easier to support multiple languages, since I wouldn't need to compile my own list of translations/labels.

This is different from -forcing- people to memorize the specific identifier in order to input it.

How do I find a specific face emoji if I don't know the name? I use the system's emoji picker tool and simply scroll through it. On OSX, it shows me recently used ones, which suffices. I don't really have to think about the names at all.

I don't see how this would change. You'd pick the cat emoji exactly the same way you do right now, and since your system language is set to English, iOS would insert :cat:, then immediately render it to an image representation.

Users wouldn't need to memorize a label any more than they currently need to memorize the Unicode positions.

Regardless however, they aren't really the hard thing for text editing.

Half the world use CJK.

[1] https://en.wikipedia.org/wiki/Pictogram

Started creating a custom text editor, then I ran into all kinds of weird difficulties like the original article mentions.

Ended up integrating Notepad++, like you integrate Word.

  · Input methods, including...  
    · Composing IME  
    · Speech  
    · Handwriting  
    · Emoji picker  
  · Complex text rendering, including...  
    · Bidi  
    · Shaping  
    · Complex-script-aware editing  
    · Complex-script-aware find, replace, etc.  
  · Accessibility

  · Advanced typography, including...  
    · About line breaking...  
      · Hyphenation  
      · Optimized line break (Knuth-plass, etc.)  
    · About microtypography  
      · OpenType features (ligatures, etc.)  
      · OpenType variations  
      · Multiple master fonts  
      · Proper font fallback ← It is not a simple lookup-at-each-character process  
    · Advanced Middle East features, including...  
      · Kashida  
    · Advanced Far East features, including...  
      · Kinsoku Shori  
      · Auto space insertion  
      · Kumimoji  
      · Warichu  
      · Ruby  
    · Proper vertical layout, including...  
      · Yoko-in-Tate  
  · Inline objects and paragraph-like objects, including...  
    · Images  
    · Hyperlinks  
    · Math equations ← This is really hard.

Text Rendering Hates You[0]

[0]: https://news.ycombinator.com/item?id=21105625

In my opinion there are several reasons why text input and selection are so difficult:

1. The behavior is complex.

2. The behavior resists being formally defined. It differs across the user interfaces, such as URL bar, rich text editor, terminal, etc.

3. The requirements are evolving. The recent big change - the emojis - is still not universally supported.

4. The reasons above tend to add accidental complexity to the API that will only grow over time.

I feel like the best approach would be to create a mathematical theory describing the text input semantics. This approach worked quite well for other complicated areas in CS, such as concurrency or memory management.

It's now clear that for someone like me - whom only understands a few Latin based languages - a super comprehensive tutorial would be required if one wanted to understand enough about other languages to be able to work on a text editor etc. I guess ideally all teams working on such projects would have an experienced team member whose native language isn't Latin based.

Speaking of input methods I always with there are good English input methods, it will be useful too! For example if user enters "compre" the input methods goes 1. comprehension 2. comprehensive (with an order depending on the conditional probability -- there is some interesting math behind input prediction, and with their help input can be a lot smooth, however I only see it used on phones not computers.)

IDEs do prediction for input.

OpenOffice used to predict words longer than say 5 characters, IIRC, it might have been back in the Star Office iteration. Come to think of it I write in a widower so little now I've only just realised that this feature might have been disabled as a default (or removed).

Most people type with Cangjie instead: https://en.wikipedia.org/wiki/Simplified_Cangjie

I can see triple-clicking+drag being tricky and that most people probably want to select entire words most of the time anyway, so I don't know what the best solution is. I'd argue that selections need handles like on mobile operating systems so it can be edited afterwards.

TextEdit and VSCode are the only editors I know of to handle transposing emoji in a reasonable way. VSCode will separate the emoji from the color modifier. TextEdit keeps them together, other editors I know of corrupt the data by slicing surrogate pairs or just no-op the command.

I disagree that the "Bad #3" example in the "Emoji Modifiers" section is actually bad though - it's the outcome I would expect of an editor.

The whole example is a bit contrived though, nobody is going to enter in a skin tone modifier character by hand in daily use. They'll select an appropriately-colored emoji.

I'll gladly put these two articles in my reading list, to go over some evening while reclined comfortably and sipping wine—so I then can laugh and slap my knees even harder when a new text editor is mentioned.