EDIT 19 October 2020: part of this post is based on faulty values. The points made remain the same, but do not rely on the exact values for further calculations.
In this series of posts, I have been tracking the effects of various manipulations on the character entropy of a given transliteration of Voynichese. This started out with practical intentions, but moved towards a more experimental approach.
Entropy Hunting and Entropy Hunting II were mainly intended to chart the effects of the popular transliteration system EVA on character entropy. For example, EVA represents the “bench” glyph as [ch], while in the manuscript it appears as one character. Since inexperienced researchers tend to assume that EVA = Voynichese, I felt like it was necessary to show the impact of these choices.
The third post, Entropy Hunting III: most frequent n-grams is similar, but with a different purpose. Here, I attempted to merge glyphs in such a way that h2 became as close to “normal” as possible, while still keeping h1 in check. While in the first posts, I tried to select the most reasonable mergers, the third post simply tries to optimize the statistics. So the question shifted from “what are caveats when using EVA?” to “what if we treat Voynichese like a verbose cipher?”.
Having moved into full-fledged cipher territory, the next step is to think about nulls (“empty” characters that are inserted to fool the reader or to pad out lines, or other characters without a sound value). My intention is not to detect which characters are most likely to be nulls – for this I refer to Nick Pelling, who seems to suspect line-initial EVA [s]. Rather, I simply wish to present the effect of various characters being nulls on entropy.
There should not necessarily be a consistant correlation between nulls and entropy – the effect depends on how the supposed nulls are distributed. For example, if I insert [0] in a text in completely random and varied places, the conditional character entropy (h2) will rise because the behavior of this new character is unpredictable. On the other hand, if I insert [0] at the beginning of every word or only after [a], predictability will increase and h2 will drop.
Let me illustrate this with an example. I took a short chapter of some 500 words from Pliny’s Natural History in Latin. Then I selected the frequency of the vowel [o] as reference: 150. For one file, I inserted 150 nulls [0] at random, making sure to position them after various letters. For the other, I inserted the numeral [0] after every [o] vowel by simply find-and-replacing [o] with [o0]. These represent two extremes: “random” nulls versus structured nulls. I am still using Nablator’s Java code for entropy calculations.
As expected, inserting 150 nulls at random raises h2 by a considerable 0.13, while consistently inserting a null after each of the 150 [o]’s lowers h2 by a similar amount.
The slippery slope of Voynich cipher thinking
Manipulating a Voynichese transliteration in order to optimize one metric, may have the undesired effect of messing up others. An obvious issue with nulls is that eliminating them would further decrease Voynichese’s character inventory (h0) which is already low. Especially when we consider only frequent characters. (Of course this is only an issue when a character is considered to be a null in all contexts, removing it entirely from the set of meaningful characters).
But if one were to go “full cipher”, then this issue might in fact solve another one. In the previous post in this series, I introduced new characters by merging frequent glyph pairs, thereby increasing h0. The effect was that characters like [o] would end up modifying the following character. So if one were to combine modifiers and nulls, then h0 might balance out.
[Side note: I don’t know what the expected h0 would be for a medieval manuscript, counting abbreviation symbols, ligatures and other positional glyph variations. This is why I usually don’t even mention the stat.]
However, this leaves us with yet another problem: word length. As Marco Ponzi wonderfully explained, my glyph merging would leave Voynichese with an average word length of 3.5, which is similar to Vietnamese and way too short for a European language. Now remove some potential nulls, and you are left with something too short for any language. Which, as usual, leads to spaces, and the question whether Voynichese words are really words at all. This is one of the reasons why I prefer to see this series of posts as experiments exploring statistics rather than potential solutions.
Pliny nulls
To get a better feel for the effect of removing characters, let us first return to Pliny’s Latin. I made a number of files where each time one character was eliminated. So Latin without [i], without [a] and so on. This allowed me to graph the impact on h2 of each removal compared to the base file.
In the above graph, we see that removing [c] decreases h2 for Latin. Apparently [c] is an unpredictable character, and removing it makes the characters more predictable overall. This appears to be the case for most characters. On the other end of the spectrum, [q,e,u,i] cause entropy to increase with their respective removal. This makes sense, since they appear in predictable combinations like [qu] and [iu].
What I wondered though, is whether I should correct for frequency. Imagine a rare glyph like [y] in Latin. Removing it will barely have any effect, even though it may be exceptionally predictable or erratic. This is why I made the next graph, by dividing the difference in h2 by the glyph’s frequency in the sample. What it shows is the increase/decrease in h2 per removed token.
This appears to work as expected, somewhat rearranging the bars and pushing glyphs with special behavior to the edges. Note how [q], probably the most predictable character, is now correctly indicated as an outlier.
Voynich nulls
How do Voynich entropy values react if we remove each glyph as if it were a null? To test this, I started with the TT transliteration and applied only two modifications: unstack benched gallows to bench-gallow and then merge the benches [ch] and [sh] to [1] and [2] respectively. The situation of [a] and [i] is more complex, so I left those unaltered.
Overall the charts for Latin and Voynichese look similar, with some glyphs lowering h2 upon removal and others raising it. However, for Latin most glyphs reduce h2, while the balance for Voynichese is more even. Moreover, the increases in h2 are much greater for Voynichese.
Below is a table with the numbers for anyone who is interested.
| count | h0 | h1 | h2 | h2/h1 | diff | diff/count | |
| base | 4.39 | 3.80 | 2.11 | 0.55 | |||
| e | 7384 | 4.32 | 3.71 | 2.12 | 0.57 | 0.014 | 0.0000019 |
| o | 6691 | 4.32 | 3.70 | 2.12 | 0.57 | 0.017 | 0.0000025 |
| a | 4799 | 4.32 | 3.69 | 2.22 | 0.60 | 0.117 | 0.0000244 |
| y | 4790 | 4.32 | 3.69 | 2.18 | 0.59 | 0.075 | 0.0000156 |
| i | 4033 | 4.32 | 3.69 | 2.16 | 0.58 | 0.049 | 0.0000121 |
| d | 3632 | 4.32 | 3.69 | 2.08 | 0.56 | -0.029 | -0.0000080 |
| ch | 3612 | 4.32 | 3.69 | 2.07 | 0.56 | -0.041 | -0.0000115 |
| k | 3446 | 4.32 | 3.69 | 2.12 | 0.58 | 0.018 | 0.0000051 |
| l | 2970 | 4.32 | 3.70 | 2.08 | 0.56 | -0.024 | -0.0000081 |
| n | 2162 | 4.32 | 3.71 | 2.18 | 0.59 | 0.074 | 0.0000341 |
| r | 2107 | 4.32 | 3.71 | 2.13 | 0.57 | 0.022 | 0.0000103 |
| q | 1880 | 4.32 | 3.71 | 2.10 | 0.57 | -0.008 | -0.0000044 |
| t | 1837 | 4.32 | 3.72 | 2.10 | 0.57 | -0.004 | -0.0000021 |
| sh | 1116 | 4.32 | 3.74 | 2.09 | 0.56 | -0.014 | -0.0000123 |
| p | 581 | 4.32 | 3.76 | 2.08 | 0.55 | -0.026 | -0.0000447 |
| s | 526 | 4.32 | 3.76 | 2.09 | 0.55 | -0.022 | -0.0000418 |
| m | 326 | 4.32 | 3.77 | 2.11 | 0.56 | 0.006 | 0.0000187 |
| f | 108 | 4.32 | 3.79 | 2.10 | 0.55 | -0.009 | -0.0000867 |
Combination
Over at the Voynich.ninja forum, Geoffrey Caveney wondered what would happen to entropy if first glyphs were combined as described in the previous post, and then [y] removed as a suspected null. As a reminder, what I did was replace the n-grams [ch, sh, ain, aiin, aiiin, air, am, ar, al, or, qok, qot, qo, ol, ot, ok, od] by a single new character. I must add again that this was already slightly more experimental than what I would usually be comfortable with.
If on top of that we assume that one of the remaining characters might be a null, the result is quite spectacular. I can best show this by adding them to the scatter plot I used in the previous post.
The grey dots represent Medieval texts in various European languages. There is a thick cluster around h2 = 3.3. But below that, between 3.0 and 3.3 there are still plenty of texts as well. The VM dots all fall within this second range. The black dot is the “original merge” version, where I tried to increase entropy by merging common n-grams. Red is OM with [d] removed. This has no effect on h2. Removing only [e] raises h2 to a respectable 3.11.
Removing only [y] performs best, raising h2 to 3.20. If you look only at h2 and h1, the VM is now a perfectly normal text, but of course there are plenty of other stats to consider, word length being a major one. Still, I must say Geoffrey was correct in assuming [y] would have the largest impact upon removal.
What this means, if anything, is another question. Entropy is not the best way to detect nulls, unless you know for sure that the nulls were applied in a very consistent way. These numbers should be treated with caution.
The Voynich Temple 
Entropy is an ineffective way of detecting nulls, as far as I can tell. The ‘signature’ of nulls is simply that they aren’t choosy who they sit next to, so their contact tables tend to be flatter than other letters’ contact table.
Incidentally, a good windmill to tilt at would be to work out whether benched gallows are more likely to be gallows+bench or bench+gallows. I have some evidence that points to gallows+bench. 🙂
LikeLike
I had been thinking about nulls earlier, but had decided not to write about them for this reason – I wouldn’t know how to detect them and entropy is mostly useless. But since Geoffrey asked about it on the forum, I decided to publish it anyway with a caveat every other paragraph 😉 Well, the stats are here should anyone ever need them.
The benched gallow problem is one I find interesting. I decided in favor of bench-gallow for two reasons:
1) If it is done in analogy with the omnipresent Greek chi-rho, then the order is small-large.
2) If benches are hypothetically as likely to appear before gallows as after them, then bench-gallow makes sense. Bench-gallow is collapsed like chi-rho, while gallow-bench remains separate. *Edit: I should perhaps specify: gallow-bench is common while the reverse is not. Unstacking as bench-gallow evens out the numbers. While there is no reason why this should be the case, I liked it more 😉
These arguments are both rather weak though, so I would be interested to learn why you prefer gallow-bench.
LikeLike
I just posted up some evidence and argument about why I think benched gallows are more likely to be enciphering gallows+ch than ch+gallows:
http://ciphermysteries.com/2020/09/28/brief-notes-on-voynichese-benched-gallows
LikeLike
An afterthought: if we know that nulls don’t tend to prefer certain neighbours, then removing them should lower h2. If they always stick to the same glyph, then removing them should raise h2. The problem is that we don’t know how the hypothetical VM nulls would be distributed.
LikeLike
Interesting that the six most predictable glyphs have been, at one time or another, suggested as variants of one another: [y] and [a], [i] and [n], [r] and [m]. A single glyph split into two due to context would make both resulting glyphs more predictable.
LikeLike
Well spotted, so this may not have been useless after all 🙂 I think you are right that we may expect a correlation between complementary distribution and predictability.
LikeLike
Thank you for performing this exercise, Koen. I appreciate it and I think the results are quite interesting.
To clarify, I did not propose [y] as a null with the goal of increasing h2 conditional entropy. I proposed [y] as a null because I find the inventory of possible vord/word/syllable endings just too excessively restricted with so many vords ending in [y]. Without [y], it actually looks much more like a natural language to me. That was my motivation to propose treating [y] as a null.
As JKP suggested on Voynich Ninja, [y] could represent a variety of grammatical suffixes and some prefixes too, as the similar-looking symbol does as an abbreviation in medieval Latin manuscripts. However, if one such symbol can represent a wide variety of grammatical suffixes, it makes it almost useless for the purpose of analyzing the structure of the underlying language. Perhaps treating it as a null is simply a useful temporary step for now, since it may represent too many different things for us to possibly make sense of them all at the present stage of investigation.
Regarding the problem of extremely small vord lengths, in addition to the idea of vords as syllables rather than words, there is also the idea that perhaps a large amount of the Voynich ms text may consist of heavily abbreviated words, in whatever language it represents. This may be a rather depressing hypothesis, since what hope do we have of deciphering a ms text full of abbreviations in a language we can’t identify with confidence? But I suspect this may be what it is. For example, what if many of the vords are abbreviated numbers in some language? For example, if it is written in a Slavic language, “pj” could clearly mean “five” to the knowledgeable reader who knows the language and knows that a number is expected in that place in the text. Such abbreviations do not even have to be consistent either: the author could variously write “pj” or “pjc” or “pjec” or other things, and any of them would still clearly mean “five” to a reader who knows the language and understands the context.
Finally, regarding the benched gallows, my own hypothesis is that they represent a palatalized form of the gallows letter, and thus each benched gallows is a single consonant phoneme unit. For example, EVA [cth] or [cph] could be the palatalized consonant “pj” that I mention above as the first consonant of the number “five” in many Slavic languages.
LikeLike
If [y] is assumed to be a universalized ending or abbreviation, then I would certainly not remove it. If I were to test this, I would rather replace all endings in the target text as well. So I would not ditch the information of [y] from the VM text, but rather try to bring the natural language text down to Voynichese level and see if it remains legible.
LikeLike
Hi, Koen: Great stuff as always, thank you! Do you think your data could be used to differentiate between a null and an abbreviation? Or does it depend on how the hypothetical null is used and what the hypothetical abbreviation replaces? At the very least, since removing did change h2, couldn’t you predict what “type” of null (e.g.predictable or random) it would have been if it was a null? Thanks again for your work,
LikeLike
It’s really just an indication of glyph behavior and how varied they like their neighbors. For example if you remove [i], you would expect an increase in entropy because it is a predictable glyph. Before it tends to come a or i, after it i, n or r. But the increase in entropy (especially when corrected for token count) is not as big as some others, because the newly created combinations are predictable as well, lots of [an] for example.
For abbreviations, I guess it depends. What kinds of abbreviations would we expect in the VM? What would their distribution look like in a normal text? That’s a whole different study.
If anything, I think your second point is possible. If you apply nulls in a specific way, the effect on entropy may be predicted, as I did when altering the Pliny text. Sprinkle nulls all around the place and entropy will go up, add them only after specific glyphs and entropy will go down.
If there are nulls in the VM, would they be the sprinkled kind or the orderly kind? Nick says: “The ‘signature’ of nulls is simply that they aren’t choosy who they sit next to”. If that is true, then removing the suspected null should lower entropy.
LikeLike
Regarding the last point about nulls and entropy: Nick’s point is correct if we are referring to nulls as typically employed in a classical cipher text. I’m not convinced the Voynich ms is in fact a cipher text at all, and when I say I treat [y] as a null I simply mean it has no phonemic value or content. For example, in the old pre-revolutionary Russian orthography, the “hard sign” had no phonemic or phonetic value or content, but it was still obligatorily written after any “hard” (non-palatalized) consonant occurring at the end of any word. (It was an artifact of Old Russian and Old Church Slavonic spelling where it did once have phonetic content as a semi-vowel.) In fact, it would be an interesting exercise in this respect to check the entropy statistics of, let’s say, a Tolstoy or Dostoevsky novel, one version published before 1917 in the old orthography (with the “null” “hard signs”) and one modern version (without the “null” “hard signs”). One could possibly do the same thing with pre-1917 and modern editions of medieval Old Russian texts.
LikeLike
I’m interested in the 9 a similarities. Because I’m analysing with a hermetic referent, I wondered if you have considered associating the letters of the alphabet on 57v with the 28 phases of the moon – waxing and waning. Don’t laugh! Arab astrologers did it and each phase received an arab phonetic sound. The middle was 15 exactly the number for our 9 symbol. Our 9 symbol is followed by c and a minim with a branch in the seventeen letter alphabet, then begins again with the o.
If you shove that c together with the other sign, you get an a in form. I wonder if it is also a 15 to begin the waning phase – and maybe the language changes? For instance, p latin-greek look the same but p is p sound in latin, r sound in greek.
I actually think I’m onto something here with the moon phases, either in arabic or adapted to latin/greek/hebrew. Your entropy would never look the same! But who knows? If you haven’t seen these Arabic charts and are interested, pm me and I’ll forward the links.
LikeLike
Barbara,
I’m interested in which system of connecting manazil and letter you’re considering. Is it Ibn Arabi’s or one of the others? As you probably know, the series can be described in differing order according to whether the counting begins from the first ‘halt’ in Orion, or the first in the constellation of Aries, or from the date of a the New Year, whether the Persian system or another.
I’d also be interested to see those charts you’ve mentioned, so perhaps if you were to give Koen permission – and if he’s not intending to republish them here – he might be kind enough to pass them on to me?.
LikeLike
Thanks for you reply. Yes, that is the chart, linked here https://classicalastrologer.me/mansions-of-the-moon-according-to-ibn-arabi/
I am not at all an astrologer but became interested because of a Kircher diagram. If the value of the ms letters changes depending on whether the moon is waxing or waning (as indicated by an o or a beginning or 9 ending, for instance), then I wondered how that would change the entropy).
I’ve seen your name pop up a few times, DN – I wonder if you could link me to your theories on the rosette page, which is my particular area of interest? Are you proposing an Arabic origin? Although I have a referent for my own semiotic analysis, the referent text itself has a complicated origin and Constantinople just popped up in that regard.
LikeLike
Barbara, thanks for your kind reply.
As to your question about my theory.. there’s no short answer so forgive a long one..
My method is not to form theories but to investigate, research in depth, and then present my conclusions from that research so I have no ‘theory’ to offer you.
About where to find my research..
Precis of my ongoing research were published in sequence between 2008-2017, in blogpost form with all the essential references and sources – primary and secondary provided to show how my eventual opinion or conclusion had been reached.
Where my conclusion was similar to some other Voynich writer’s, I credited those of whom I learned, whether or not I’d seen their work before conducting my own study. This courtesy of acknowledgement makes it easier for those who come later as well as avoiding any appearance of theft and the presence of fastidious credit for precedents and sources serves also as a handy way to distinguish trained from amateur Voynich writers.
However by 2017 I had grown tired of seeing quite so much of my original research – or rather my conclusions and observations – being re-presented without that basic courtesy of the precedent study being acknowledged so in that year I finally closed off from the public the work I’d published online. Some of it is still being recycled, inevitably without the informing sources and the evidence which makes sense of the conclusion itself. The ‘re-cyclers’ habitually represent those conclusions as if ‘just ideas’ of their own. Not helpful in the long run, but I’m happy to say that Koen is among the people who simply don’t do such things.
As far as public writings are concerned since 2017, I’ve concentrated on writing a critical review of the history of ‘Voynich studies’. Intended for the more serious sort of scholar, its basic plan is to provide a brief background to some issue or past ideas about some section of the manuscript and then make a short list of basic or essential readings likely to serve as a first stage of researching that matter, while highlighting points where the ‘standard tale’ about this manuscript fails to meet basic criteria in method, proof, current thinking in historical studies and so on.
That blog is called
Voynich Revisionst – a bibliography
Of my research conclusions to date …
I’ve never offered them as a “theory” in either sense of the term. This is because – first, there’s nothing ‘theoretical’ about conclusions reached after fairly in-depth research into a question and, secondly, I don’t have any neat, overarching theory-of-everything which makes a snappy sentence.
But had this difficult and fascinating manuscript been amenable to an easy intuitive (or imaginative) ‘theory’, it would surely have been sorted out pretty well before World War II, by which time most scripts and languages, as well as most styles of drawing, could be accurately identified on sight by specialists in manuscripts and keepers of the great collections.
One observation I would like to make, since you seem to be newly come to the study.
It is my observation, over more than a decade, that the chief pitfalls for novices are:
To adopt too easily the older ideas which may be seen everywhere, but which generally pass by conferred, rather than merited ‘authority’. Some of these, when traced to their source, are found to be no more than the imagination of Wilfrid Voynich – they are not (and never would be) modern assessments of the manuscript.
Also, in my earnest opinion, any researcher errs who fails to distinguish on the one hand between the manuscript’s time and place of manufacture (discovering as much as possible of both), and on the other the time(s) and place(s) which saw the contained matter given its first enunciation. As illustration – you may have in your hand a fifteenth-century French bible, but it is a gave mistake to simply presume that the text within was first composed in France, or in the fifteenth century.
Carelessness on that score is not uncommon in Voynich writings, as a cavalier attitude towards what defines ‘similarity’ between forms of script or ‘paired’ pictures. Too often in the enthusiasm of creating plausible ‘ads’ for a theory, the theorists create such parallels without much attention or study of the relevant discipline. One has to be on guard against the facile and the enthusiast whose chief interest is his own ‘theory’ .
So , at last..
thank you most sincerely for replying. On a side-note. The author of the blog from which you had the chart once kindly gave – and bravely allowed me to quote – his correction of one common, if erroneous, theory about the month-diagrams.
Regards
LikeLike
Thank you, Diane! I have an MA, I’ve been published, and I taught students citation fundamentals in English Lit so anything I formally publish should meet your standards of academic rigor. Mind you, that was years ago so I’ll have to brush up on online standards.
It is unfortunate you felt you must withdraw your findings. Nobody is very interested in mine online – as soon as I mention Hermetica they withdraw (though I had never heard of the Hermetica until this summer – I am not a believer in or practioner of any kind of esoteric spirituality, or any mainstream kind, for that matter ) but I will nevertheless be publishing an article because I feel my findings are significant to the context of illustrations and the text itself.
No one, as far as I know, has been looking at the text glyphs as symbols in and of themselves and how they relate to the rosettes page and its symbol sets, nor set out a cohesive pattern of relationships so that our linguists and cryptographers might actually have a chance of solving the text. My expertise is in symbolic narrative and culture, so despite my inexperience with the VMS ( though 4 months intensive reading and Voychinese symbols and letters haunting my nightmares can scarcely be called complete inexperience), I do hope you’ll read it and enjoy it as a good effort if nothing else. Moreover, it is funny! Once you understand the symbols, the illustrations abound in metaphor and puns.
By the way, if you do know of anyone who has studied the rosettes page through a comprehensive, philisophical and technical, Hermetic lens, please do let me know.
We are soooo off-topic! Please forgive, Koen. Diane did me such a courtesy with her long reply, I felt the need to reply in kind. Diane, meet you over on Koen’s most recent Entropy blog.
LikeLike