Difference between revisions of "Transition probabilities from selected texts"

From Derek
Jump to: navigation, search
(→‎Full Text)
(More languages and wider variety of texts for existing languages)
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
(Denley: Sorry about the CSV but I don't have the patience to format it into a table. Maybe sometime I'll get java to do it for me. If u copy it to a text file and call it a .csv u can open it in excel)
 
−
 
 
The Somerton Man's code (without the extra line) is 44 characters long. So, if the text is purely random (1/26 chance of each letter appearing) then the probability of attaining this particular string of 44 is (1/26)^44 = 5.51027E-63. This is a good initial comparison.
 
The Somerton Man's code (without the extra line) is 44 characters long. So, if the text is purely random (1/26 chance of each letter appearing) then the probability of attaining this particular string of 44 is (1/26)^44 = 5.51027E-63. This is a good initial comparison.
  
−
==Full Text==
+
For transitions that have p=0, corrections to p=0.0001 have been performed to attain a non-zero Markov probability.
−
 
+
−
===English (Orwell)===
+
−
 
+
−
Probability of the Somerton Man's code coming from this text: 1.708E-67
+
−
(One of the transitions had zero probability so perhaps zero?; I counted log0 = 0)
+
−
 
+
−
,A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z
+
−
A,0.18%,2.88%,4.80%,6.43%,0.08%,1.22%,2.55%,0.47%,4.31%,0.06%,1.59%,9.08%,2.63%,16.65%,0.13%,2.96%,0.09%,10.28%,13.23%,13.26%,0.86%,2.12%,1.30%,0.05%,2.63%,0.15%
+
−
B,6.37%,1.36%,0.05%,0.07%,34.19%,0.03%,0.03%,0.13%,3.87%,0.69%,0.01%,13.77%,0.22%,0.05%,10.98%,0.00%,0.00%,8.32%,1.35%,0.51%,11.15%,0.46%,0.18%,0.00%,6.20%,0.00%
+
−
C,11.99%,0.15%,1.76%,0.11%,18.11%,0.11%,0.04%,14.99%,4.95%,0.00%,6.27%,3.34%,0.06%,0.03%,19.56%,0.12%,0.05%,5.31%,0.33%,8.84%,3.15%,0.06%,0.16%,0.00%,0.52%,0.00%
+
−
D,8.79%,4.76%,1.70%,2.82%,12.51%,2.31%,1.25%,4.65%,12.04%,0.51%,0.21%,2.17%,1.65%,3.48%,9.61%,1.25%,0.09%,2.80%,7.13%,11.09%,2.38%,0.46%,4.03%,0.00%,2.33%,0.00%
+
−
E,7.67%,1.82%,3.83%,9.43%,4.55%,2.34%,1.05%,2.57%,2.81%,0.13%,0.45%,4.14%,3.75%,9.60%,2.71%,2.82%,0.22%,13.70%,9.09%,6.71%,0.53%,2.49%,4.55%,1.05%,1.95%,0.03%
+
−
F,10.45%,1.33%,2.03%,0.92%,9.58%,5.61%,0.65%,3.94%,9.78%,0.17%,0.23%,3.76%,1.13%,0.59%,15.52%,1.96%,0.04%,7.77%,2.52%,14.77%,3.58%,0.44%,1.89%,0.00%,1.33%,0.01%
+
−
G,11.40%,2.28%,1.04%,0.95%,13.60%,1.55%,1.55%,16.80%,8.35%,0.13%,0.05%,3.32%,0.89%,1.85%,8.95%,0.99%,0.04%,5.79%,4.58%,8.15%,3.91%,0.23%,2.74%,0.00%,0.83%,0.02%
+
−
H,18.87%,0.26%,0.31%,0.16%,47.30%,0.30%,0.10%,0.90%,14.60%,0.03%,0.02%,0.26%,0.26%,0.25%,7.05%,0.20%,0.03%,1.17%,0.64%,4.67%,1.03%,0.04%,0.77%,0.00%,0.77%,0.00%
+
−
I,2.18%,0.97%,5.85%,4.47%,3.51%,2.25%,2.90%,0.14%,0.04%,0.00%,0.95%,4.17%,5.26%,30.46%,4.40%,0.77%,0.06%,3.15%,10.74%,14.69%,0.08%,1.97%,0.18%,0.26%,0.00%,0.56%
+
−
J,4.81%,0.00%,0.00%,0.00%,21.88%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.22%,0.00%,0.00%,18.38%,0.00%,0.00%,0.00%,0.00%,0.00%,54.70%,0.00%,0.00%,0.00%,0.00%,0.00%
+
−
K,6.05%,1.55%,1.03%,0.58%,31.18%,1.42%,0.36%,3.39%,18.38%,0.03%,0.06%,2.22%,0.83%,11.99%,3.41%,0.75%,0.14%,0.25%,5.14%,4.05%,1.28%,0.17%,4.41%,0.00%,1.33%,0.00%
+
−
L,8.28%,0.89%,0.67%,8.60%,17.34%,2.25%,0.24%,0.80%,12.90%,0.04%,0.75%,13.32%,1.36%,0.33%,8.24%,0.85%,0.02%,0.77%,3.29%,4.16%,1.83%,0.56%,1.69%,0.00%,10.84%,0.00%
+
−
M,15.21%,3.95%,0.37%,0.26%,29.76%,0.55%,0.24%,1.30%,10.95%,0.04%,0.04%,0.33%,2.17%,0.54%,12.82%,6.21%,0.02%,0.88%,3.53%,3.90%,3.52%,0.05%,1.75%,0.00%,1.60%,0.00%
+
−
N,5.25%,1.02%,3.64%,13.99%,10.02%,1.33%,13.77%,2.49%,5.15%,0.10%,1.07%,1.55%,0.62%,1.15%,9.26%,0.73%,0.17%,0.46%,6.94%,15.56%,1.00%,0.49%,2.25%,0.02%,1.95%,0.02%
+
−
O,1.22%,2.15%,1.94%,2.24%,0.51%,10.95%,0.95%,1.05%,1.43%,0.05%,1.42%,3.74%,6.26%,14.97%,3.74%,2.22%,0.02%,11.27%,3.71%,7.54%,14.85%,1.84%,5.19%,0.13%,0.55%,0.06%
+
−
P,15.92%,0.43%,0.19%,0.09%,19.67%,0.37%,0.07%,2.89%,6.16%,0.01%,0.01%,9.32%,0.20%,0.28%,12.70%,6.97%,0.00%,11.16%,2.94%,5.40%,3.72%,0.00%,0.79%,0.00%,0.70%,0.00%
+
−
Q,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,100.00%,0.00%,0.00%,0.00%,0.00%,0.00%
+
−
R,8.97%,1.11%,1.53%,3.36%,25.68%,1.23%,1.04%,2.46%,9.52%,0.07%,1.19%,1.75%,2.21%,1.96%,10.52%,1.08%,0.07%,2.19%,6.56%,8.67%,2.18%,0.48%,1.75%,0.00%,4.40%,0.00%
+
−
S,10.01%,1.88%,2.89%,0.95%,10.50%,1.75%,0.49%,7.07%,8.42%,0.13%,0.56%,1.99%,2.28%,2.19%,8.84%,3.54%,0.22%,0.62%,7.56%,20.04%,3.03%,0.32%,3.87%,0.00%,0.87%,0.00%
+
−
T,5.77%,1.09%,1.01%,0.58%,8.96%,0.75%,0.25%,35.18%,8.88%,0.06%,0.23%,1.62%,0.89%,0.53%,12.03%,0.61%,0.05%,3.47%,3.54%,5.84%,1.86%,0.08%,4.09%,0.00%,2.63%,0.00%
+
−
U,4.07%,2.23%,4.29%,1.86%,3.11%,0.67%,6.39%,0.82%,2.26%,0.03%,0.30%,12.89%,3.02%,10.05%,0.53%,4.73%,0.00%,13.67%,11.20%,16.29%,0.13%,0.12%,1.08%,0.06%,0.14%,0.05%
+
−
V,6.97%,0.00%,0.00%,0.00%,72.86%,0.00%,0.00%,0.00%,12.40%,0.00%,0.00%,0.00%,0.00%,0.02%,6.76%,0.00%,0.00%,0.07%,0.00%,0.00%,0.09%,0.00%,0.02%,0.00%,0.81%,0.00%
+
−
W,29.02%,0.28%,0.37%,0.53%,13.65%,0.12%,0.09%,16.54%,17.10%,0.02%,0.08%,0.72%,0.51%,3.84%,10.59%,0.16%,0.01%,1.58%,1.84%,1.70%,0.08%,0.03%,0.91%,0.00%,0.23%,0.00%
+
−
X,11.63%,0.38%,13.65%,0.38%,7.08%,0.38%,0.25%,2.15%,19.85%,0.00%,0.00%,1.26%,1.26%,0.00%,1.01%,15.30%,0.13%,0.00%,0.51%,18.20%,3.03%,0.00%,1.26%,0.00%,2.28%,0.00%
+
−
Y,8.57%,3.58%,3.61%,2.29%,7.57%,3.05%,1.07%,5.07%,6.48%,0.15%,0.47%,1.85%,3.40%,1.39%,18.96%,2.08%,0.13%,1.87%,8.94%,10.51%,0.77%,0.34%,6.68%,0.00%,1.15%,0.01%
+
−
Z,11.76%,0.00%,0.00%,0.00%,57.84%,0.00%,0.00%,0.00%,13.07%,0.00%,0.00%,2.94%,0.00%,0.00%,6.86%,0.00%,0.00%,0.00%,0.00%,0.00%,0.33%,0.65%,0.00%,0.00%,1.63%,4.90%
+
−
 
+
−
===French (Hugo - Les Orientales)===
+
−
 
+
−
Probability of the Somerton Man's code coming from this text: 8.769E-71
+
−
 
+
−
Accented Characters were mapped to a character from A-Z. ie. é -> E
+
  
−
,A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z
+
HMMER score<ref>ftp://selab.janelia.org/pub/software/hmmer/CURRENT/Userguide.pdf Page 43</ref> is the log (base 2) of Markov probability / null probability (1/26^44)
−
A,0.21%,2.46%,3.95%,2.70%,0.26%,1.42%,2.82%,0.83%,14.15%,0.32%,0.05%,7.26%,4.72%,19.68%,0.20%,2.77%,0.83%,9.37%,6.17%,4.95%,10.34%,3.78%,0.03%,0.03%,0.38%,0.33%
+
−
B,16.55%,0.08%,0.00%,0.15%,17.76%,0.00%,0.00%,0.08%,5.87%,0.00%,0.00%,18.89%,0.00%,0.00%,11.29%,0.00%,0.00%,26.71%,0.30%,0.00%,1.88%,0.08%,0.00%,0.00%,0.38%,0.00%
+
−
C,7.84%,0.11%,1.11%,0.77%,21.54%,0.00%,0.08%,22.95%,5.81%,0.04%,0.08%,4.86%,0.23%,0.11%,21.84%,0.11%,0.19%,5.93%,2.41%,0.96%,2.72%,0.04%,0.00%,0.00%,0.19%,0.08%
+
−
D,14.77%,0.24%,0.36%,1.33%,47.64%,0.12%,0.09%,0.57%,7.70%,0.30%,0.00%,0.73%,0.42%,0.24%,8.13%,0.36%,0.21%,3.05%,3.26%,0.36%,9.79%,0.24%,0.00%,0.06%,0.00%,0.00%
+
−
E,5.83%,1.33%,3.92%,3.61%,6.00%,1.93%,1.09%,0.39%,2.32%,0.69%,0.02%,6.52%,3.47%,9.43%,1.01%,2.77%,1.10%,7.35%,21.28%,8.58%,7.78%,2.78%,0.00%,0.12%,0.06%,0.62%
+
−
F,14.00%,0.08%,0.00%,0.08%,15.25%,6.41%,0.00%,0.00%,9.54%,0.08%,0.00%,18.30%,0.08%,0.00%,11.88%,0.00%,0.16%,14.86%,1.49%,0.23%,7.58%,0.00%,0.00%,0.00%,0.00%,0.00%
+
−
G,9.50%,0.00%,0.38%,0.75%,28.41%,0.09%,0.00%,0.66%,5.64%,0.00%,0.00%,8.65%,0.28%,8.09%,4.42%,0.09%,0.19%,18.44%,1.60%,1.98%,10.07%,0.19%,0.00%,0.00%,0.56%,0.00%
+
−
H,32.20%,0.29%,0.29%,0.00%,40.18%,0.00%,0.00%,0.00%,7.59%,0.19%,0.00%,0.58%,0.68%,0.10%,10.60%,0.29%,0.88%,1.46%,0.29%,0.39%,2.72%,0.29%,0.00%,0.00%,0.97%,0.00%
+
−
I,0.93%,1.08%,2.36%,3.12%,14.19%,1.17%,2.17%,0.12%,0.79%,0.43%,0.02%,16.20%,3.05%,12.11%,0.93%,1.60%,1.00%,8.46%,14.50%,11.68%,0.05%,2.43%,0.00%,1.21%,0.00%,0.41%
+
−
J,20.70%,0.00%,0.00%,0.00%,41.61%,0.00%,0.00%,0.00%,2.18%,0.00%,0.00%,0.00%,0.00%,0.00%,30.07%,0.00%,0.00%,0.00%,0.00%,0.00%,5.45%,0.00%,0.00%,0.00%,0.00%,0.00%
+
−
K,0.00%,0.00%,0.00%,5.00%,20.00%,0.00%,0.00%,5.00%,15.00%,0.00%,0.00%,15.00%,0.00%,0.00%,20.00%,0.00%,0.00%,10.00%,10.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%
+
−
L,19.89%,0.45%,0.58%,1.17%,43.12%,0.71%,0.24%,1.04%,4.40%,0.18%,0.00%,10.97%,0.44%,0.50%,8.01%,0.47%,0.78%,0.24%,1.80%,0.97%,3.56%,0.34%,0.00%,0.00%,0.13%,0.00%
+
−
M,15.19%,10.11%,0.11%,0.26%,32.61%,0.00%,0.07%,0.00%,6.94%,0.00%,0.00%,0.34%,9.96%,0.34%,13.40%,6.49%,0.07%,0.00%,0.49%,0.04%,3.17%,0.11%,0.00%,0.00%,0.30%,0.00%
+
−
N,4.66%,1.02%,6.62%,8.87%,13.63%,3.16%,3.93%,0.32%,2.15%,0.67%,0.05%,1.40%,0.70%,2.91%,5.39%,1.62%,1.34%,1.10%,12.40%,23.49%,2.43%,1.89%,0.00%,0.05%,0.08%,0.13%
+
−
O,0.14%,0.50%,1.15%,0.61%,1.29%,0.36%,0.32%,0.30%,12.65%,0.23%,0.00%,3.96%,9.02%,23.91%,0.02%,0.79%,0.23%,9.90%,3.10%,3.14%,27.22%,0.07%,0.00%,0.00%,1.06%,0.00%
+
−
P,23.41%,0.05%,0.00%,0.30%,16.69%,0.05%,0.00%,1.68%,7.16%,0.05%,0.00%,14.17%,0.15%,0.00%,15.06%,2.52%,0.10%,10.37%,2.67%,1.33%,3.95%,0.00%,0.00%,0.00%,0.30%,0.00%
+
−
Q,0.00%,0.00%,0.00%,0.00%,0.00%,0.10%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,99.90%,0.00%,0.00%,0.00%,0.00%,0.00%
+
−
R,11.15%,1.65%,2.72%,5.96%,25.21%,0.76%,1.29%,0.17%,7.12%,0.39%,0.00%,4.96%,2.88%,1.43%,9.39%,1.28%,1.04%,3.34%,9.85%,5.57%,3.08%,0.72%,0.00%,0.02%,0.00%,0.02%
+
−
S,9.20%,2.11%,5.20%,7.26%,15.84%,2.68%,0.93%,0.79%,3.65%,1.04%,0.02%,6.15%,3.12%,1.44%,8.41%,4.55%,2.93%,1.53%,9.41%,6.19%,4.67%,2.31%,0.00%,0.12%,0.46%,0.00%
+
−
T,9.10%,0.99%,3.03%,6.46%,20.31%,0.89%,0.45%,0.61%,4.18%,0.71%,0.02%,7.80%,1.44%,1.08%,9.92%,2.80%,2.00%,8.42%,10.82%,4.44%,3.29%,0.96%,0.00%,0.12%,0.07%,0.09%
+
−
U,2.32%,1.34%,2.12%,2.22%,9.66%,1.51%,0.99%,0.20%,11.98%,0.92%,0.03%,4.66%,2.22%,10.50%,0.89%,2.67%,0.53%,19.56%,8.41%,5.75%,0.60%,2.90%,0.00%,7.75%,0.21%,0.05%
+
−
V,18.18%,0.00%,0.00%,0.06%,32.42%,0.00%,0.00%,0.00%,18.88%,0.00%,0.00%,0.38%,0.19%,0.06%,24.48%,0.00%,0.00%,4.58%,0.06%,0.00%,0.70%,0.00%,0.00%,0.00%,0.00%,0.00%
+
−
W,100.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%
+
−
X,5.96%,4.43%,8.41%,11.01%,5.50%,3.21%,1.99%,1.22%,4.13%,0.76%,0.00%,4.89%,5.66%,1.38%,2.29%,7.65%,4.43%,3.52%,5.66%,4.74%,0.61%,5.66%,0.00%,5.05%,1.83%,0.00%
+
−
Y,18.55%,0.90%,3.62%,3.17%,38.91%,0.45%,0.00%,0.00%,0.00%,0.00%,0.90%,2.26%,2.26%,0.45%,6.33%,4.98%,0.00%,9.05%,6.33%,0.90%,0.00%,0.90%,0.00%,0.00%,0.00%,0.00%
+
−
Z,20.48%,0.60%,1.81%,4.22%,10.24%,0.00%,0.00%,0.00%,3.61%,0.60%,0.00%,5.42%,3.61%,0.00%,23.49%,1.81%,0.00%,1.20%,1.20%,0.60%,6.02%,13.86%,0.00%,0.00%,0.00%,1.20%
+
  
−
==Initial Letters==
+
<!--
 +
This software output has been formatted in html for quick entry into the project wiki
 +
-->
 +
==First order==
 +
===All letters===
 +
<br/>(..\Texts\English.txt) All Letters
 +
<br/>Markov Probability: 4.196215910162246E-70
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        -23.646530132315654
 +
<br/>
 +
<br/>(..\Texts\French.txt) All Letters
 +
<br/>Markov Probability: 4.562440416695874E-74
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        -36.8135257053655
 +
<br/>
 +
<br/>(..\Texts\German.txt) All Letters
 +
<br/>Markov Probability: 1.6093650169064557E-82
 +
<br/>Corrected Zeroes:    3
 +
<br/>HMMER Score:        -64.89226460456342
 +
<br/>
 +
<br/>(..\Texts\Spanish.txt) All Letters
 +
<br/>Markov Probability: 1.7633169297716054E-83
 +
<br/>Corrected Zeroes:    1
 +
<br/>HMMER Score:        -68.08239247668342
 +
<br/>
 +
<br/>(..\Texts\Italian.txt) All Letters
 +
<br/>Markov Probability: 2.8938109466376915E-92
 +
<br/>Corrected Zeroes:    9
 +
<br/>HMMER Score:        -97.26506645809364
 +
<br/>
 +
<br/>(..\Texts\Portuguese.txt) All Letters
 +
<br/>Markov Probability: 1.0172950778991843E-77
 +
<br/>Corrected Zeroes:    1
 +
<br/>HMMER Score:        -48.94437749826932
 +
<br/>
 +
<br/>(..\Texts\Dutch.txt) All Letters
 +
<br/>Markov Probability: 2.139309827314818E-84
 +
<br/>Corrected Zeroes:    1
 +
<br/>HMMER Score:        -71.12546693519374
 +
<br/>
 +
<br/>(..\Texts\Swedish.txt) All Letters
 +
<br/>Markov Probability: 4.7024882053115854E-77
 +
<br/>Corrected Zeroes:    2
 +
<br/>HMMER Score:        -46.735691382905166
 +
<br/>
 +
<br/>(..\Texts\Vigenere - 1984.txt) All Letters
 +
<br/>Markov Probability: 1.646391769425068E-70
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        -24.99631136880728
 +
<br/>
 +
<br/>(Outputs\Playfair.out) All Letters
 +
<br/>Markov Probability: 5.213910076344393E-69
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        -20.01132524791302
 +
<br/>
 +
===Initial letters===
 +
<br/>(..\Texts\English.txt) Initial Letters
 +
<br/>Markov Probability: 5.755746003335865E-56
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        23.316377212971148
 +
<br/>
 +
<br/>(..\Texts\French.txt) Initial Letters
 +
<br/>Markov Probability: 1.960919656262944E-61
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        5.153264236029422
 +
<br/>
 +
<br/>(..\Texts\German.txt) Initial Letters
 +
<br/>Markov Probability: 7.441017498436695E-73
 +
<br/>Corrected Zeroes:    1
 +
<br/>HMMER Score:        -32.78590341697
 +
<br/>
 +
<br/>(..\Texts\Spanish.txt) Initial Letters
 +
<br/>Markov Probability: 3.204888821639082E-63
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        -0.7818480694202504
 +
<br/>
 +
<br/>(..\Texts\Italian.txt) Initial Letters
 +
<br/>Markov Probability: 6.089831612262369E-59
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        13.431992337096512
 +
<br/>
 +
<br/>(..\Texts\Portuguese.txt) Initial Letters
 +
<br/>Markov Probability: 2.2658166834014361E-60
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        8.683693049158332
 +
<br/>
 +
<br/>(..\Texts\Dutch.txt) Initial Letters
 +
<br/>Markov Probability: 6.9654365323502316E-68
 +
<br/>Corrected Zeroes:    1
 +
<br/>HMMER Score:        -16.27154908302583
 +
<br/>
 +
<br/>(..\Texts\Swedish.txt) Initial Letters
 +
<br/>Markov Probability: 2.6806903224847996E-64
 +
<br/>Corrected Zeroes:    2
 +
<br/>HMMER Score:        -4.361445908011734
 +
<br/>
 +
==Second order==
 +
===All letters===
 +
<br/>(..\Texts\English.txt) All Letters
 +
<br/>Markov Probability: 4.2148901763982914E-92
 +
<br/>Corrected Zeroes:    9
 +
<br/>HMMER Score:        -96.72254209077907
 +
<br/>
 +
<br/>(..\Texts\French.txt) All Letters
 +
<br/>Markov Probability: 1.9814441750241465E-90
 +
<br/>Corrected Zeroes:    9
 +
<br/>HMMER Score:        -91.16762862009702
 +
<br/>
 +
<br/>(..\Texts\German.txt) All Letters
 +
<br/>Markov Probability: 5.919358467581905E-105
 +
<br/>Corrected Zeroes:    14
 +
<br/>HMMER Score:        -139.41766153806188
 +
<br/>
 +
<br/>(..\Texts\Spanish.txt) All Letters
 +
<br/>Markov Probability: 3.342953425806875E-98
 +
<br/>Corrected Zeroes:    11
 +
<br/>HMMER Score:        -116.98848244535708
 +
<br/>
 +
<br/>(..\Texts\Italian.txt) All Letters
 +
<br/>Markov Probability: 6.083262400057097E-116
 +
<br/>Corrected Zeroes:    21
 +
<br/>HMMER Score:        -175.9194661728711
 +
<br/>
 +
<br/>(..\Texts\Portuguese.txt) All Letters
 +
<br/>Markov Probability: 4.2738731323579313E-94
 +
<br/>Corrected Zeroes:    13
 +
<br/>HMMER Score:        -103.34634923820269
 +
<br/>
 +
<br/>(..\Texts\Dutch.txt) All Letters
 +
<br/>Markov Probability: 5.327306011536052E-112
 +
<br/>Corrected Zeroes:    17
 +
<br/>HMMER Score:        -162.82319287449383
 +
<br/>
 +
<br/>(..\Texts\Swedish.txt) All Letters
 +
<br/>Markov Probability: 1.1823873333660746E-91
 +
<br/>Corrected Zeroes:    10
 +
<br/>HMMER Score:        -95.23440631711085
 +
<br/>
 +
<br/>(..\Texts\Vigenere - 1984.txt) All Letters
 +
<br/>Markov Probability: 1.669944098510842E-92
 +
<br/>Corrected Zeroes:    8
 +
<br/>HMMER Score:        -98.05823732223358
 +
<br/>
 +
<br/>(Outputs\Playfair.out) All Letters
 +
<br/>Markov Probability: 7.389612924665265E-86
 +
<br/>Corrected Zeroes:    6
 +
<br/>HMMER Score:        -75.98096976556741
 +
<br/>
 +
===Initial letters===
 +
<br/>(..\Texts\English.txt) Initial Letters
 +
<br/>Markov Probability: 1.0496288884966237E-55
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        24.18318171171002
 +
<br/>
 +
<br/>(..\Texts\French.txt) Initial Letters
 +
<br/>Markov Probability: 8.83846402382603E-70
 +
<br/>Corrected Zeroes:    3
 +
<br/>HMMER Score:        -22.571823368605646
 +
<br/>
 +
<br/>(..\Texts\German.txt) Initial Letters
 +
<br/>Markov Probability: 1.6757717490935368E-89
 +
<br/>Corrected Zeroes:    10
 +
<br/>HMMER Score:        -88.08742718870606
 +
<br/>
 +
<br/>(..\Texts\Spanish.txt) Initial Letters
 +
<br/>Markov Probability: 8.869612541473453E-71
 +
<br/>Corrected Zeroes:    3
 +
<br/>HMMER Score:        -25.888676055317255
 +
<br/>
 +
<br/>(..\Texts\Italian.txt) Initial Letters
 +
<br/>Markov Probability: 1.0949986922100798E-57
 +
<br/>Corrected Zeroes:    0
 +
<br/>HMMER Score:        17.600375336401736
 +
<br/>
 +
<br/>(..\Texts\Portuguese.txt) Initial Letters
 +
<br/>Markov Probability: 9.986807259493189E-70
 +
<br/>Corrected Zeroes:    4
 +
<br/>HMMER Score:        -22.395595515749598
 +
<br/>
 +
<br/>(..\Texts\Dutch.txt) Initial Letters
 +
<br/>Markov Probability: 1.9749729570078271E-69
 +
<br/>Corrected Zeroes:    2
 +
<br/>HMMER Score:        -21.41185805018986
 +
<br/>
 +
<br/>(..\Texts\Swedish.txt) Initial Letters
 +
<br/>Markov Probability: 9.55816081723446E-69
 +
<br/>Corrected Zeroes:    4
 +
<br/>HMMER Score:        -19.13695790770939
 +
<br/>
  
−
===English (Orwell)===
 
  
−
Note: not a single word in the text starts with an X hence the "NaN"
+
==References==
 +
<References/>
  
−
Probability of the Somerton Man's code deriving from this text: 1.922E-56
+
==See also==
 +
*[[Cipher Cracking 2009]]
 +
*[[Markov models]]
  
−
,A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z
+
==Back==
−
A,8.97%,3.97%,4.20%,3.26%,2.70%,4.82%,1.62%,6.60%,5.59%,0.31%,0.61%,3.25%,3.71%,2.20%,4.96%,3.89%,0.42%,2.43%,8.02%,18.05%,0.99%,1.10%,6.91%,0.00%,1.41%,0.01%
+
*[https://myuni.adelaide.edu.au/webapps/portal/frameset.jsp Back to MyUni]
−
B,12.60%,4.66%,3.39%,3.13%,2.03%,3.53%,0.94%,7.24%,8.57%,0.38%,0.64%,1.69%,2.21%,2.13%,7.55%,2.69%,0.26%,1.71%,6.82%,18.20%,1.42%,0.84%,6.18%,0.00%,1.16%,0.02%
+
*[http://www.eleceng.adelaide.edu.au/personal/dabbott Back to Derek Abbott's homepage]
−
C,10.47%,6.53%,2.50%,1.88%,2.12%,2.97%,1.37%,6.76%,8.18%,0.26%,0.21%,1.11%,2.32%,4.64%,11.95%,2.30%,0.13%,1.68%,5.83%,16.18%,0.83%,0.26%,7.95%,0.00%,1.60%,0.00%
+
*[http://www.eleceng.adelaide.edu.au Back to EEE Department page]
−
D,11.84%,5.04%,2.18%,1.76%,1.90%,3.31%,1.02%,6.73%,9.83%,0.14%,0.85%,1.73%,2.18%,7.68%,8.24%,1.44%,0.14%,1.41%,4.37%,15.46%,1.02%,0.28%,8.31%,0.00%,3.13%,0.00%
+
*[http://www.adelaide.edu.au Back to the University of Adelaide homepage]
−
E,13.01%,4.56%,3.28%,2.24%,3.12%,4.60%,1.20%,6.61%,9.33%,0.36%,0.64%,1.72%,2.32%,1.68%,8.57%,2.36%,0.08%,1.56%,5.28%,16.33%,1.08%,0.52%,8.61%,0.00%,0.92%,0.00%
+
−
F,14.94%,3.53%,2.69%,2.14%,2.55%,3.15%,0.95%,7.24%,6.37%,0.33%,0.22%,1.57%,3.39%,1.11%,9.95%,2.01%,0.14%,0.95%,5.80%,20.26%,1.11%,0.38%,7.19%,0.00%,2.03%,0.00%
+
−
G,13.62%,4.54%,2.58%,1.89%,1.47%,3.28%,1.05%,7.75%,7.05%,0.35%,0.14%,1.47%,2.65%,1.26%,9.92%,3.00%,0.14%,1.75%,6.49%,15.71%,2.93%,0.42%,8.52%,0.00%,2.03%,0.00%
+
−
H,8.34%,8.95%,5.34%,2.91%,2.57%,4.31%,1.56%,10.10%,4.29%,0.36%,1.37%,2.37%,3.80%,3.19%,4.89%,2.50%,0.15%,2.09%,9.58%,9.62%,0.93%,0.72%,9.42%,0.00%,0.64%,0.00%
+
−
I,10.62%,2.64%,2.79%,2.40%,1.95%,2.62%,1.21%,7.60%,7.25%,0.18%,0.31%,1.04%,2.83%,3.44%,4.51%,2.52%,0.06%,1.21%,5.75%,20.66%,0.82%,0.46%,15.30%,0.00%,1.85%,0.00%
+
−
J,19.65%,4.40%,2.05%,1.17%,1.76%,3.23%,0.59%,7.04%,8.50%,0.29%,0.29%,2.64%,1.47%,2.05%,8.50%,3.81%,0.29%,1.17%,7.92%,10.56%,0.88%,0.00%,11.73%,0.00%,0.00%,0.00%
+
−
K,11.40%,2.34%,0.73%,1.17%,1.75%,1.02%,0.15%,7.46%,9.94%,0.15%,0.29%,0.44%,1.90%,1.02%,15.50%,0.88%,0.00%,0.58%,3.65%,22.22%,0.58%,0.29%,14.47%,0.00%,2.05%,0.00%
+
−
L,18.13%,5.55%,2.45%,2.78%,1.94%,3.52%,1.02%,5.37%,7.40%,0.37%,0.28%,1.67%,2.78%,2.08%,8.93%,2.22%,0.19%,1.34%,6.38%,14.11%,1.71%,0.28%,8.05%,0.00%,1.48%,0.00%
+
−
M,11.78%,6.57%,4.40%,2.29%,1.66%,3.38%,0.96%,9.34%,7.32%,0.24%,0.48%,1.99%,2.26%,2.26%,11.75%,2.77%,0.12%,1.24%,5.94%,12.18%,1.15%,0.60%,7.87%,0.00%,1.42%,0.03%
+
−
N,9.53%,5.20%,3.52%,4.63%,5.47%,4.06%,1.42%,6.47%,6.12%,0.19%,1.84%,3.10%,4.71%,1.84%,7.42%,2.33%,0.46%,3.21%,6.97%,11.98%,1.22%,0.54%,6.89%,0.00%,0.84%,0.04%
+
−
O,10.09%,3.64%,4.10%,2.23%,2.47%,2.90%,1.28%,8.49%,4.97%,0.34%,0.62%,2.06%,3.53%,1.44%,6.65%,4.07%,0.16%,1.50%,5.66%,25.28%,0.55%,0.82%,5.96%,0.00%,1.18%,0.01%
+
−
P,11.57%,4.34%,2.94%,1.98%,2.19%,3.18%,0.61%,6.30%,9.39%,0.29%,0.12%,1.69%,3.21%,1.78%,13.55%,2.94%,0.32%,1.31%,4.95%,14.22%,1.84%,0.15%,9.97%,0.00%,1.17%,0.00%
+
−
Q,14.49%,3.74%,1.87%,4.21%,4.21%,1.40%,1.87%,5.14%,6.54%,0.47%,0.00%,5.14%,1.40%,0.93%,15.89%,3.74%,0.00%,3.27%,5.61%,9.81%,1.40%,0.93%,7.48%,0.00%,0.47%,0.00%
+
−
R,12.15%,5.25%,2.01%,2.16%,2.16%,3.76%,0.88%,9.01%,9.47%,0.46%,0.31%,1.39%,1.70%,2.01%,11.07%,1.54%,0.15%,0.98%,4.48%,19.21%,1.29%,0.26%,7.36%,0.00%,0.88%,0.05%
+
−
S,12.41%,4.20%,2.62%,2.85%,1.82%,3.15%,1.05%,7.44%,7.60%,0.51%,0.53%,1.91%,3.03%,1.58%,11.26%,2.58%,0.28%,1.48%,5.92%,15.77%,1.55%,0.56%,8.44%,0.00%,1.48%,0.00%
+
−
T,6.49%,5.36%,4.93%,3.49%,2.77%,4.10%,2.23%,7.40%,4.77%,0.33%,0.77%,2.48%,4.61%,1.78%,5.09%,6.72%,0.15%,3.10%,8.34%,12.05%,0.97%,0.66%,9.81%,0.00%,1.57%,0.01%
+
−
U,16.61%,3.55%,2.84%,1.51%,1.69%,3.37%,0.62%,8.79%,8.53%,0.00%,0.09%,1.51%,2.31%,1.33%,4.97%,1.24%,0.09%,1.15%,4.97%,24.07%,0.98%,0.18%,8.61%,0.00%,0.98%,0.00%
+
−
V,9.64%,4.08%,4.74%,1.96%,4.08%,7.03%,2.45%,3.43%,5.56%,0.65%,0.49%,4.08%,4.25%,1.96%,9.15%,1.96%,0.33%,1.96%,8.01%,10.78%,2.61%,0.82%,9.15%,0.00%,0.82%,0.00%
+
−
W,13.78%,4.74%,4.10%,2.94%,2.48%,2.91%,1.47%,8.41%,7.18%,0.45%,0.77%,2.09%,2.16%,4.98%,5.57%,2.48%,0.29%,1.58%,7.52%,13.48%,1.02%,0.44%,7.52%,0.00%,1.62%,0.00%
+
−
X,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN
+
−
Y,11.51%,3.84%,7.68%,3.29%,2.33%,3.36%,1.78%,10.08%,4.73%,0.41%,2.47%,3.29%,3.84%,2.12%,5.48%,1.78%,0.07%,1.92%,6.51%,9.12%,1.37%,0.34%,11.45%,0.00%,1.23%,0.00%
+
−
Z,25.00%,0.00%,0.00%,0.00%,0.00%,12.50%,0.00%,0.00%,12.50%,0.00%,0.00%,0.00%,0.00%,0.00%,25.00%,12.50%,0.00%,0.00%,0.00%,0.00%,0.00%,0.00%,12.50%,0.00%,0.00%,0.00%
+

Latest revision as of 16:48, 2 October 2009

The Somerton Man's code (without the extra line) is 44 characters long. So, if the text is purely random (1/26 chance of each letter appearing) then the probability of attaining this particular string of 44 is (1/26)^44 = 5.51027E-63. This is a good initial comparison.

For transitions that have p=0, corrections to p=0.0001 have been performed to attain a non-zero Markov probability.

HMMER score[1] is the log (base 2) of Markov probability / null probability (1/26^44)

First order

All letters


(..\Texts\English.txt) All Letters
Markov Probability: 4.196215910162246E-70
Corrected Zeroes: 0
HMMER Score: -23.646530132315654

(..\Texts\French.txt) All Letters
Markov Probability: 4.562440416695874E-74
Corrected Zeroes: 0
HMMER Score: -36.8135257053655

(..\Texts\German.txt) All Letters
Markov Probability: 1.6093650169064557E-82
Corrected Zeroes: 3
HMMER Score: -64.89226460456342

(..\Texts\Spanish.txt) All Letters
Markov Probability: 1.7633169297716054E-83
Corrected Zeroes: 1
HMMER Score: -68.08239247668342

(..\Texts\Italian.txt) All Letters
Markov Probability: 2.8938109466376915E-92
Corrected Zeroes: 9
HMMER Score: -97.26506645809364

(..\Texts\Portuguese.txt) All Letters
Markov Probability: 1.0172950778991843E-77
Corrected Zeroes: 1
HMMER Score: -48.94437749826932

(..\Texts\Dutch.txt) All Letters
Markov Probability: 2.139309827314818E-84
Corrected Zeroes: 1
HMMER Score: -71.12546693519374

(..\Texts\Swedish.txt) All Letters
Markov Probability: 4.7024882053115854E-77
Corrected Zeroes: 2
HMMER Score: -46.735691382905166

(..\Texts\Vigenere - 1984.txt) All Letters
Markov Probability: 1.646391769425068E-70
Corrected Zeroes: 0
HMMER Score: -24.99631136880728

(Outputs\Playfair.out) All Letters
Markov Probability: 5.213910076344393E-69
Corrected Zeroes: 0
HMMER Score: -20.01132524791302

Initial letters


(..\Texts\English.txt) Initial Letters
Markov Probability: 5.755746003335865E-56
Corrected Zeroes: 0
HMMER Score: 23.316377212971148

(..\Texts\French.txt) Initial Letters
Markov Probability: 1.960919656262944E-61
Corrected Zeroes: 0
HMMER Score: 5.153264236029422

(..\Texts\German.txt) Initial Letters
Markov Probability: 7.441017498436695E-73
Corrected Zeroes: 1
HMMER Score: -32.78590341697

(..\Texts\Spanish.txt) Initial Letters
Markov Probability: 3.204888821639082E-63
Corrected Zeroes: 0
HMMER Score: -0.7818480694202504

(..\Texts\Italian.txt) Initial Letters
Markov Probability: 6.089831612262369E-59
Corrected Zeroes: 0
HMMER Score: 13.431992337096512

(..\Texts\Portuguese.txt) Initial Letters
Markov Probability: 2.2658166834014361E-60
Corrected Zeroes: 0
HMMER Score: 8.683693049158332

(..\Texts\Dutch.txt) Initial Letters
Markov Probability: 6.9654365323502316E-68
Corrected Zeroes: 1
HMMER Score: -16.27154908302583

(..\Texts\Swedish.txt) Initial Letters
Markov Probability: 2.6806903224847996E-64
Corrected Zeroes: 2
HMMER Score: -4.361445908011734

Second order

All letters


(..\Texts\English.txt) All Letters
Markov Probability: 4.2148901763982914E-92
Corrected Zeroes: 9
HMMER Score: -96.72254209077907

(..\Texts\French.txt) All Letters
Markov Probability: 1.9814441750241465E-90
Corrected Zeroes: 9
HMMER Score: -91.16762862009702

(..\Texts\German.txt) All Letters
Markov Probability: 5.919358467581905E-105
Corrected Zeroes: 14
HMMER Score: -139.41766153806188

(..\Texts\Spanish.txt) All Letters
Markov Probability: 3.342953425806875E-98
Corrected Zeroes: 11
HMMER Score: -116.98848244535708

(..\Texts\Italian.txt) All Letters
Markov Probability: 6.083262400057097E-116
Corrected Zeroes: 21
HMMER Score: -175.9194661728711

(..\Texts\Portuguese.txt) All Letters
Markov Probability: 4.2738731323579313E-94
Corrected Zeroes: 13
HMMER Score: -103.34634923820269

(..\Texts\Dutch.txt) All Letters
Markov Probability: 5.327306011536052E-112
Corrected Zeroes: 17
HMMER Score: -162.82319287449383

(..\Texts\Swedish.txt) All Letters
Markov Probability: 1.1823873333660746E-91
Corrected Zeroes: 10
HMMER Score: -95.23440631711085

(..\Texts\Vigenere - 1984.txt) All Letters
Markov Probability: 1.669944098510842E-92
Corrected Zeroes: 8
HMMER Score: -98.05823732223358

(Outputs\Playfair.out) All Letters
Markov Probability: 7.389612924665265E-86
Corrected Zeroes: 6
HMMER Score: -75.98096976556741

Initial letters


(..\Texts\English.txt) Initial Letters
Markov Probability: 1.0496288884966237E-55
Corrected Zeroes: 0
HMMER Score: 24.18318171171002

(..\Texts\French.txt) Initial Letters
Markov Probability: 8.83846402382603E-70
Corrected Zeroes: 3
HMMER Score: -22.571823368605646

(..\Texts\German.txt) Initial Letters
Markov Probability: 1.6757717490935368E-89
Corrected Zeroes: 10
HMMER Score: -88.08742718870606

(..\Texts\Spanish.txt) Initial Letters
Markov Probability: 8.869612541473453E-71
Corrected Zeroes: 3
HMMER Score: -25.888676055317255

(..\Texts\Italian.txt) Initial Letters
Markov Probability: 1.0949986922100798E-57
Corrected Zeroes: 0
HMMER Score: 17.600375336401736

(..\Texts\Portuguese.txt) Initial Letters
Markov Probability: 9.986807259493189E-70
Corrected Zeroes: 4
HMMER Score: -22.395595515749598

(..\Texts\Dutch.txt) Initial Letters
Markov Probability: 1.9749729570078271E-69
Corrected Zeroes: 2
HMMER Score: -21.41185805018986

(..\Texts\Swedish.txt) Initial Letters
Markov Probability: 9.55816081723446E-69
Corrected Zeroes: 4
HMMER Score: -19.13695790770939


References

  1. ↑ ftp://selab.janelia.org/pub/software/hmmer/CURRENT/Userguide.pdf Page 43

See also

Back

Retrieved from "https://www1.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php?title=Transition_probabilities_from_selected_texts&oldid=4087"