Editing Cracking the Voynich code 2014

== Supervisors ==
*[[Derek Abbott|Prof Derek Abbott]]
*[[Brian W.-H. Ng|Dr Brian Ng]]
*[[Maryam Ebrahimpour]]

==Honours students==
*[[Bryce Shi]]
*[[Peter Roush]]

==Project guidelines==
*[http://www.eleceng.adelaide.edu.au/personal/dabbott/project_handbook_2009.pdf Project Handbook]

==General project description==
The Voynich Manscript is a mysterious 15th century book that no one today know what it says or who wrote it.   The book is in a strange alphabet. See details [https://en.wikipedia.org/wiki/Voynich_manuscript here].

Fortunately the whole book has been converted into an electronic format with each character changed to a convenient ascii character.  We want you to write software that will search the text and perform statistical tests to get clues as to the nature of the writing.  Does the document bear the statistics of a natural language or is it a fake?

We already have Support Vector Machine (SVM) and Multiple Discriminant Analysis (MDA) software that you can adapt for your purposes.  This software is set up to test if two texts are written by the same author or not. The great thing about our software is that it is independent of language.   So you could compare it against the existing writings of Roger Bacon, who is a suspected author

==Specific tasks==

Here are the remaining tasks resulting from previous work. You may want to focus on a subset of these:

* Critically review the statistical analysis of the letters. See if you can extend it (eg. testing another language previous students missed and checking if they included all possibilities of ambiguous letters).  Is the conclusion of previous team correct?  Because there are lots of interesting new tasks (see below) don't spend too long on this.  Just spend enough time on this as a quick warm up exercise.

* Download the searchable pdf file of the copy of the [http://www.eleceng.adelaide.edu.au/personal/dabbott/tamanshud/W&T_rubaiyat_wells_copy.pdf Omar Khayyam] that closely matches the dead man's copy. Create an ascii file with the raw text. Use this as a one-time pad that directly substitutes the letters of the alphabet a-z. The book contains 75 quatrains (four sentence poems) each containing about 140 letters.  So the whole book contains about <math>75 \times 140 = 10,500\,</math> letters. As we don't know where in the book the one-time pad starts, start at the beginning and step through the whole book one letter at a time. You'll then end up with >10,000 decrypts. Write a software script to look for the most common top-20 words in all the decrypts to narrow down to a few possible results that can be examined by eye.

* Extend the [[Media:Cipher GUI.rar|CipherGUI 2011]] software that was created by a previous team.  See if you can add more ciphers to the collection. Use it to eliminate more ciphers and enter your conclusions here: [[Cipher Cross-off List]]. Be critical and be prepared to question and recheck some of the items already on the list.

* A previous team created a webcrawler and search engine to search keywords with wild cards, as Google does not allow this. This is  to check for common repeated expressions on the WWW that may contain initial letters that are also in the code. If the code is an initialism, this will give us a clue as to some likely content. There are two things that need to be fixed: (i) we need you to write a convenient web-interface for this search engine, and (ii) we need to scrap the webcrawler as it takes too long. It is impossible to crawl the whole web with one little PC and therefore you need to interface the search engine to operate on an index that has already been created by a commercial webtrawler. Unfortunately, companies like Google won't let you access their index list.  Therefore you need to use another provider such as [http://yacy.de/en/index.html YaCy]. You may also want to check out a resource called [http://commoncrawl.org/get-started/ CommonCrawl]. To use CommonCrawl you'll need to sign up with both [https://commoncrawl.atlassian.net/wiki/display/CRWL/Quick+Start+-+Build+from+Github github] (to download the Java source code) and also [http://docs.aws.amazon.com/ElasticMapReduce/latest/GettingStartedGuide/Welcome.html?r=4524 Amazon] (to run your uploaded compiled code). 

* Use computer graphics to reconstruct and undistort the face of the dead man. What would he look like if he were alive? To do this you need the data from the previous group that scanned the man's face from a plaster bust, at the Police Museum, with a [http://www.david-laserscanner.com/ 3D scanner]. An example of the type of graphics software you can use to manipulate the scanned image is [http://www.123dapp.com/catch 123D]. You may want to investigate other 3D rendering graphics software.

* Use the departmental 3D printer to recreate a scaled down version of the bust, before and after your 3D rendering. The motivation for creating a 3D representation is so that we can create 2D pictures of the man at any angle. It will not be long before companies like Google release next generation search engines that search for faces on the web. So having multiple images at a number of angles will be of future importance for a large-scale image search.

* Investigate the prices and availability of more expensive 3D scanners than the [http://www.david-laserscanner.com/ David 3D Scanner] that we have. Can you find a 3D scanner that would have the resolution to pick up all the pores and texture on the bust of the Somerton Man?

* Plot, present, and interpret the mass spectrometer data we have.

==Deliverables==

===Semester A===

* Proposal seminar (Thursday or Friday, Week 4)
* Progress report (Friday, Week 12) - only one report needed in wiki format

===Semester B===

* Final seminar (Friday, 1st week of mid-semester break)
* Final report (Friday, Week 11) - only one report needed in wiki format
* Poster (Tuesday, Week 12) - one poster only needed
* Project exhibition 'expo'  (Friday, Week 12) 
* CD or stick containing your whole project directories (Tuesday, Week 13)
* YouTube video (Tuesday, Week 13) - add the URL to this wiki

== Weekly progress and questions ==
This is where you record your progress and ask questions. Make sure you update this every week.
*[[Cipher cracking 2013 weekly progress]]

==Approach and methodology==
We expect you to take a structured approach to both the validation of last year's results, and the writing of the software. You should carefully design the big-picture high-level view of the software modules, and the relationships and interfaces between them. Think also about the data transformations needed.

==Possible extension== 
If you knock off this project too easily and are looking for a harder code cracking problem to try your software out on, you can progress to analyzing another famous unsolved mystery: the [http://en.wikipedia.org/wiki/Voynich_manuscript Voynich Manuscript]

== Expectations == 
We don't really expect you to find the killer, though that would be cool if you do and you'll become very famous overnight. To get good marks we expect you to show a logical approach to trying to find the patterns from the code on the web, and any other attempts to crack the code. Running the webcrawler for many hours is unrealistic, so we'd like you to find a pre-indexed version of the web for you search engine.

== Relationship to possible career path==
Whilst the project is fascinating as you'll learn about a specific murder case—and we do want you to have a lot of fun with it—the project does have a hard-core serious engineering side. It will familiarize you with techniques in information theory, probability, statistics, encryption, decryption, and datamining. It will also improve your software skills. The project will also involve writing software code that trawls for patterns on the world wide web (exploiting it as a huge database). This will force you to learn about search engines and databases; and the new tools you develop may lead to new IP in the area of datamining and also make you rich/famous. The types of jobs out there where these skills are useful are in computer security, comms, or in digital forensics. The types of industries that will need you are: the software industry, e-finance industry, e-security, IT industry, Google, telecoms industry, [http://www.asio.gov.au/ ASIO], [http://www.asis.gov.au/ ASIS], defence industry (e.g. [http://www.dsd.gov.au/ DSD]), etc. So go ahead and have fun with this, but keep your eye on the bigger engineering picture and try to build up an appreciation of why these techniques are useful to our industry. Now go find that killer...this message will self-destruct in five seconds :-)

==See also==
* [[Cipher_Cracking_2013/Rubaiyat_Plaintext]]
* [[Cipher Cracking]] (main page)
* [[Cipher cracking 2013 weekly progress]]
* [[Semester A Progress Report 2013 - Cipher cracking]]
* [[Semester B Final Report 2013 - Cipher cracking]]
* [[Cipher Cross-off List]]
* [[File:Cipher_GUI.rar]]
* [[File:DecodingToolkit.rar]]

== References and useful resources==
If you find any useful external links, list them here:
* [http://en.wikipedia.org/wiki/Taman_Shud_Case The taman shud case]
* [http://yacy.de/en/index.html YaCy]
* [http://commoncrawl.org/get-started/ CommonCrawl].
* [http://www.library.cornell.edu/colldev/mideast/okhym.htm Edward Fitzgerald's translation of رباعیات عمر خیام by  عمر خیام]
* [http://ebooks.adelaide.edu.au/ Adelaide Uni Library e-book collection]
* [http://www.gutenberg.org/wiki/Main_Page Project Gutenburg e-books]
* [http://onlinebooks.library.upenn.edu/archives.html#foreign Foreign language e-books]
* [http://www.ohchr.org/EN/UDHR/Pages/Introduction.aspx UN Declaration of Human Rights - different languages]
* [http://www.eleceng.adelaide.edu.au/personal/dabbott/tamanshud/SSC_mckay1999.pdf Statistical debunking of the 'Bible code']
* [http://enc.slider.com/Enc/OneTimePads One time pads]
* [http://www.fbi.gov/hq/lab/fsc/backissu/jan2000/olson.htm Analysis of criminal codes and ciphers]
* [http://www.fbi.gov/hq/lab/fsc/backissu/april2006/research/2006_04_research01.htm Code breaking in law enforcement: A 400-year history]
* [http://portal.acm.org/citation.cfm?id=1389095.1389425 Evolutionary algorithm for decryption of monoalphabetic homophonic substitution ciphers encoded as constraint satisfaction problems]

==Back==
*[https://myuni.adelaide.edu.au/webapps/portal/frameset.jsp Back to MyUni]
*[http://www.eleceng.adelaide.edu.au/personal/dabbott Back to Derek Abbott's homepage]
*[http://www.eleceng.adelaide.edu.au Back to EEE Department page]
*[http://www.adelaide.edu.au Back to the University of Adelaide homepage]
*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/Cipher_Cracking_2012 Back to Cipher Cracking 2012 homepage]