Difference between revisions of "Progress Report 2010"

From Derek
Jump to: navigation, search
Line 1: Line 1:
 
==Due Date==
 
==Due Date==
  
****edit****
+
edit****
  
==Abstract==
+
==Executive Summary==
This project explores the meaning behind a secret message that is related to the dead body of an unknown man found at Somerton beach in 1948. The main purpose of the project is to create a web crawler to specifically search for words and patterns that match or are similar to the Somerton man's code. A java based open source web crawler will be used, and will be rewritten and modified in order to perform the searches that we need to acquire our results.
+
For decades the written code that is linked to the “Summerton Man Mystery” (or Taman Shud Case) has been undecipherable and delivered no solid information about his history or identity. This project aims to bring light to the unsolved mystery by tapping into the wealth of information available on the internet.
  
==Aim==
+
Recent collection of random letter samples has been used to reinforce previous claims that the code is unlikely to just be a randomly generated sequence of letters. Along with thorough revision and checking of previous code used to produce results, it has been agreed that the Summerton Man code is most likely initialism.
The project involves using engineering techniques such as information theory, probability, statistics, encryption, decryption and software writing skills. Using these techniques we attempt to recreate and validate the results of the project from past years in order to better understand what the secret message is. The Somerton man's code is thought to be an initialism of a sentence or text. The main aim of the project is to attempt to create a web crawler based in java code to find patterns in text on websites. Using these patterns, we hope to use frequency analysis to create a list of potential words that the initialism could stand for.
+
  
==Background and Significance==
+
The code found with the dead man will be used in conjunction with a suitable “Web Crawler” and specific pattern matching algorithms. The aim is to target websites of interest and filter through the raw text available in search of character patterns that are also present in the code.  Once some solid results are obtained, frequency analysis will hopefully enable a solid hypothesis on some possibilities in regards to the meaning behind the code.
  
 +
unfinished***
  
===The Victim===
 
  
[[Image:Suitcase.jpg|thumb|170px|right|Suitcase and effects, found at Adelaide Railway Station.]]
 
The body of an unknown man was found resting against a rock wall on Somerton Beach opposite a crippled children's home shortly before 6:45 am on the 1st of December in 1948. A post-mortem of the body revealed that there was no possibility that the death was natural due to severe congestion in his organs. The most likely cause of death was from poison, which in 1994 was uncovered by forensic science to be digitalis<ref name="Taman Shud Wiki">http://en.wikipedia.org/wiki/Taman_Shud_Case</ref>.
 
  
Among the man’s possessions were cigarettes, matches, a metal comb, chewing gum, a railway ticket to Henley Beach, a bus ticket and a tram ticket. A brown suitcase belonging to the man was later discovered at the Adelaide Railway Station. In it contained various articles of clothing, a screwdriver, a stencilling brush, a table knife, scissors and other such utensils. Authorities found the name “T. Keane” and the unknown man was temporarily identified as a local sailor by the name of Tom Keane. However, this was proved to be incorrect and the man’s identity has been a mystery ever since. The Somerton man has since been incorrectly identified as a missing stable hand, a worker on a steamship, and a Swedish man<ref name="Taman Shud Wiki"/>.
+
==Aims and Objectives==
 +
The aim of the project is to make progression with respect to the Summerton Man Case through use of engineering techniques such as information theory, problem solving, statistics, encryption, decryption and software coding. Through use of such techniques, the results from the previous year’s project will revised and used to solidify the pathway for the rest of the project. Once past results are properly revised the next aim of the project is to attempt to utilise a web crawling device in order to access information stored in raw text on websites of interest. Specifically designed algorithms will then be used on the sourced text in order to search for patterns and reoccurrences that may be of interest or relation to the contents of the Summerton Man Code.
  
A spy theory also began circulating due to the circumstances of the Somerton man’s death. In 1947, The US Army’s Signal Intelligence Service, as part of Operation Venona, discovered that there had been top secret material leaked from Australia’s Department of External Affairs to the Soviet embassy in Canberra. Three months prior to the death of the Somerton man, on the 16th of August 1948, an overdose of digitalis was reported as the cause of death for US Assistant Treasury Secretary Harry Dexter White, who had been accused of Soviet espionage under Operation Venona. Due to the similar causes of their death, the rumours of the unknown man being a Soviet spy began to spread.
+
==Project Background==
  
===The Code===
+
==Requirements and Specifications==
The most puzzling of his belongings was a slip of paper bearing the words “Tamam Shud”, which translated to “ended” or “finished”. The fragment of paper was identified as being from the last page of a book of poems called [http://en.wikipedia.org/wiki/Rubaiyat_of_Omar_Khayyam The Rubiayat by Omar Khayyam], a famous Persian poet.
+
include block diagrams/workload breakdown
  
A police announcement was made through the media searching for a copy of The Rubiayat with the “Tamam Shud” phrase removed. Shortly afterward, a local person came forward to reveal that he had found a rare first edition of the book in the back seat of his unlocked car in Glenelg on the 30th of November, 1948.
+
==Progress So Far==
  
Inside that particular copy of The Rubiayat was a phone number penciled in on the back cover which led police to a nurse known as Jestyn (her real name is withheld). The woman has since denied all knowledge of the dead man. Also penciled into the book, was a short code.
+
==Approach for remaining part of the project==
  
  
[[Image:Somertoncode.jpg|thumb|left|The handwriting of the Somerton Man showing his pencil markings in the back of a book of poetry by Omar Khayyam.]]
 
 
The code has been unsolvable for some time due to the quality of hand writing. The first character on the first and third line looks like an “M” or “W”, and the fifth line’s first character looks like an “I” or a “V”. The second line is crossed out (and is omitted entirely in previous cracking attempts), and there is an “x” above the “O” on the fourth line. Due to the ambiguity of some of these letters and lines, some possibly wrong assumptions could be made as to what is and isn’t a part of the code.
 
 
 
Professional attempts at unlocking this code were largely limited due to the lack of modern techniques and strategies, because they were carried out decades earlier. When the code was analysed by experts in the Australian Department of Defence in 1978, they made the following statements regarding the code:
 
 
* There are insufficient symbols to provide a pattern.
 
* The symbols could be a complex substitute code or the meaningless response to a disturbed mind.
 
* It is not possible to provide a satisfactory answer.
 
 
 
 
Last year’s [[Final report 2009: Who killed the Somerton man?|code cracking project group]] ran a number of tests to selectively conclude which encryption method was mostly likely used. In their final report, they determined that the Somerton man’s code was not a transposition or substitution cipher. They tested the Playfair Cipher, the Vigenere Cipher and the one-time pad<ref>[[Final report 2009: Who killed the Somerton man?]]</ref>. The group concluded that the code was most likely to be an initialism of a sentence of text and using frequency analysis, determined that the most likely intended sequence of letters in the code was:
 
 
 
 
<center>WRGOABABD</center>
 
<center>WTBIMPANETP</center>
 
<center>MLIABOAIAQC</center>
 
<center>ITTMTSAMSTCAB </center>
 
 
===Significance===
 
Probably since the beginning of recognizable human behaviour, coding has been fundamental to all human groups. It was a way to enable communication when ordinary spoken or written language was difficult or impossible. But for as long as communication was needed, concealment, too, seems to be just as common in human societies. Secret languages and gestures are characteristic of many human groups, serving as a means of concealing messages.
 
 
A cipher system is the systematic concealment of a message by substituting words or phrases with a different set of characters. They began to be widely used in the 16th century, but they date back as far as 6th century BC<ref name="Code Book">The Secrets of Codes: Understanding the World of Hidden Messages by Paul Lunde</ref>.
 
 
[[Image:Caesarshift.jpg|thumb|250px|right|A three letter [http://en.wikipedia.org/wiki/Caesar_cipher Caesar Shift]. There are 26 possible shifts in the English language]]
 
Coding and cipher systems have historically been limited to espionage or warfare, but the recent transformation of communication technology has added new opportunities and challenges for the budding cryptologist. We live in an age, with computerised communication, where the most valuable commodity is code and the second most valuable is the private information that it gives access to. In order to guard this information, we use encryption systems that have to be continually updated as every encryption system is eventually foiled. How to preserve privacy in an electronic age is one of the burning questions of our time<ref name="Code Book"/>.
 
 
In this day and age, engineers now have the knowledge and skills of information theory and statistics to better analyse the data of previously unsolvable codes. We believe that the cracking of this code holds the key to the identity of the Somerton man. Solving this mystery could potentially bring closure to the man’s family as well as others who have become obsessed with the case.
 
 
As stated, the main goal of the project is to create a java based web crawler. This web crawler is planned to be quite versatile and have many applications outside of this project. For instance, it could be used for further cipher or code related searching not exclusive to just the Somerton man’s code. Also, the web crawler would be able to search for different text patterns and language comparisons, or even to identify plagiarism. Furthermore, it could be used as a web searcher like Google with perhaps additional functions such as being able to search with wildcard letters (ie you could search for “be*r” where the * is a wildcard, or any letter).
 
 
==Requirements==
 
 
===Key Requirements===
 
The key requirements for the project are shown below. They have been determined by the group in order to fully utilize the web crawler so that we can obtain the best results possible.
 
 
The code must handle getting an initial URL to start the web search, as well as getting subsequent URLs thereafter. Whether this is done by choosing a URL at random or a specific list is currently unknown/undecided.
 
 
The code must handle sub-links or web pages within a main page. That is, web pages with multiple frames or source codes. It is important that our code can handle this because if it doesn’t, precious data can be missed and left unparsed or unrecorded.
 
 
There must be special cases when errors occur. This includes Input/Output errors that cause the web page to fail loading as well as invalid URL errors. For both cases, we have to decide whether to re-attempt parsing the page, or to simply skip it. Moreover, another special case must be considered for non-HTML sources such as PDF files or Microsoft Word documents. We should parse these because the data is probably useful for our results. Raw data programs should probably be skipped because it will mostly be code and not helpful.
 
 
Additionally, the code must be able to determine the difference between HTML code and raw text and data. The HTML code will be largely useless and probably repetitive in multiple sites which wouldn’t be useful for the results we are after. The information we need comes from the raw text and data which we need to parse. However, the comments in the HTML code may be useful samples for our results; we may choose to parse them, but if we are not able to parse comments while ignoring other HTML code, it should be skipped.
 
 
The code must be able to parse both single characters and just initial letters of a word. This is mostly for added functionality, because it has been concluded that the Somerton man’s code is probably an initialism. This is done because we want our web crawler to have uses outside of this project. How our code chooses the initial letter of a word is also critical. In most cases it is simply the character following a space, but there will also have to be special cases (like words following a period with incorrect punctuation, such as “hello.there").
 
 
Moreover, we want the code to be able to search for exact patterns as well as “similar” patterns. That is, data that has the same pattern as our given input, but with different characters (for example, if our input is searching for “ABAB” and comes across text with “XYXY”, it would be considered as a “similar” pattern and noted down). Also, the code must handle non-letter characters such as punctuation.
 
 
The most important requirement is probably what the code does with the results. For every “found” pattern, the code should put the text (or sentence or phrase that the pattern was found in) and URL in some sort of list or array. This is significant because it will provide the basis of our results. From this, we can get frequency logs of certain words to better determine what the initialisms in the Somerton man’s code could possibly be.
 
 
===Coding Requirements===
 
The code must be well documented and commented. Clear explanations of our code will let others understand it more easily. This is an advantage in terms of maintenance and modification.
 
 
If we want our code to act as a standalone program, it must also have a well defined interface. The program should be able to switch between parsing single characters or initial letters and exact or similar patterns with the press of a button.
 
 
The code is to be written in java because that is the language that the group is most comfortable with. It is probably one of the most common coding languages available so it can be easily maintained and modified. It is also free to download and very accessible.
 
 
==Proposed Approach==
 
The project is essentially broken up into two main components.  These are:
 
#. Reviewing and verifying past results. 
 
#. Implementing a web crawling device to search the internet for items of interest. 
 
 
Briefly reviewing the techniques and procedures used by the previous year’s students in their statistical analysis of the Somerton Man Code will allow for a clearer understanding of the exact requirements of the second component of the project. It is currently expected that the Web Crawler will be searching for patterns of repetition in text both in sequence and as initialism (where each letter of the code is the first letter of a word). However, if the results obtained this year are significantly different to the previous outcome, the web crawler may be used to search for some other item of interest.
 
 
 
[[Image:Random_letter_sample.jpg|thumb|170px|left|Sample sheet used to collect random samples from straight and intoxicated subjects]]
 
===Preliminary work carried out so far===
 
 
Previous statistical analysis on the mystery code has shown that the letters in the code are most likely some type of English Initialism. This was achieved by running a series of tests and comparisons on the code to eliminate possibilities and find the most probable outcome.
 
 
To verify that the code is not just random "meaningless" letters, random letter samples are being compiled from a range of different subjects (male, female, below 25 years old, above 25 years old). The random samples are also being collected from subjects who have been exposed to some level of alcohol.  This will allow for comparison between the mystery code and a true sample of random letters in order to determine the likelihood that the code has been purposely randomised as an act of deception or intoxication.  To keep the samples as realistic as possible it has been decided that each person will list 40 random letters.  There are about 45 letters on the Summerton Man Code and it has already been noticed that after about 10 random letters most subjects slow down and start thinking more about the letters they are writing.  The results will ideally reflect the characteristics of the mystery code, if it is in fact random.
 
 
Verifying the accuracy of the results previously obtained with respect to the language of the code and the probability of initialism involves thoroughly checking the software used to run the past tests.  This software is currently being revised and checked in order to accept its performance.  Initial proof reading and error checking has given good evidence that the software used was in fact working properly.
 
 
===Approach for remaining part of the project===
 
 
====Verification of Past Results====
 
 
Once the previous software has been thoroughly checked and re-run, and collection of random letter samples is completed and compiled, the verification of previous results will be completed.  This leaves two possibilities for the direction of the project:
 
 
*Past results are satisfied.
 
 
In this case the web crawler will be designed to search websites for patterns in text.  The patterns of interest will be specific to those that occur in the Somerton Man Code.
 
 
*Past results are not satisfied.
 
 
In this case the result will be followed by new conclusions with respect to the characteristics of the Somerton Man Code.  Extensive testing and error checking will then be undertaken to satisfactorily prove the new conclusions.  The use of the web crawler will reflect the new results and be designed to search websites for whatever item of interest will possibly provide answers to the meaning of the Code.
 
 
====Development of the Web Crawler====
 
 
Ideally a Java based web crawler will be implemented.  Both members of the project team are most comfortable with Java and its operation.  Two Java based web crawlers that are currently being researched and experimented with are J-spider<ref>J-spider Web Crawler http://j-spider.sourceforge.net/</ref> and Arachnid <ref>Arachnid Web Crawler http://arachnid.sourceforge.net/</ref>.  Learning the techniques available with these crawlers and their individual ways of searching the web is an involved process and rather steep learning curve.  One risk associated with this approach is if a suitable Java based web crawler is not available.  This would involve sourcing out another language crawler that will allow for the suitable searching to be undertaken.  Becoming familiar with another programming language will take time and would not be ideal for the project schedule.  This is not a huge issue at the present time as the Java web crawlers being looked at appear to have characteristics that are of interest to the project, and there are many other Java based crawlers that have not been investigated yet.
 
 
The Web Crawler will be designed to follow a set of predefined rules in order to traverse a web site and report on its findings with respect to inputs provided by the user.  To simplify the construction of the Web Crawler, it will be developed in several smaller modules.
 
 
* Produce a simple Java program/algorithm for traversing HTML code.  The program will only search through raw text and ignore all of the unwanted syntax that is found in HTML code such as the frequently occurring parenthesis.  Once this has been achieved it will be necessary to evolve the algorithm to take inputs from a user defining what item is to be searched for.  The program will also be able to identify links on web pages and store them for future use.
 
 
 
* Become familiar with a specific web crawling device and produce a successful search.  This search will be as simple as searching through a singular web page and indicating that the entire page was traversed with no errors.  Once this simple search has been completed more complex searching can be adapted such as finding links and traversing a linked web page.
 
 
 
*Merge the searching algorithm with the simple web crawling device.  The Crawler will firstly be used on individual websites to search and report on its findings.
 
 
 
An initial design for the software can be seen in the block diagram below.
 
[[Image:Software_block_diagram.jpg|600px|centre]]
 
 
 
As can be seen in the block diagram the general operation of the software is reasonably straight forward and simple. The basic operation will go as follows:
 
 
* Get input from a user
 
* Go to the starting Web page
 
* Run through a series of checks such as:
 
:::*If at end of page
 
:::*If a link has been found
 
:::*If a pattern has been found
 
* When a pattern is found it will be saved somewhere along with information or relevant details of its location
 
* When a link is found it will be added to a queue so that it can be used later
 
* When the bottom of a page is reached the whole process will begin from the top of the page that is associated with the link at the front of the link queue
 
* Once the link queue is empty (ie there are no more pages to traverse) the process is finished
 
* Ideally all patterns found along with their relevant information will be saved in a file
 
  
 
==Project Management==
 
==Project Management==
===Milestones & Work Breakdown===
 
The entire project has been divided into smaller elements.  This allows for more efficient organisation and management of both resources and time.  By breaking each task into smaller modules it becomes much easier to allocate different tasks to each team member whilst also keeping track of the project progress.  The breakdown of the project can be seen in the Gantt Chart below.
 
 
 
[[Image:cipher_gantt.jpg|1200px|centre]]
 
 
 
There are several tasks that can be completed in parallel.  Being in a group of 2 has meant that wherever possible there is only a maximum of two tasks running at once.  This allows for project tasks to run in parallel without overloading on one group member resulting in delayed or rushed products.  Having tasks in parallel makes for much more efficient use of time and significantly reduces the risk of confusion through 2 people working on one task.  The only times when there are more than the ideal amount of tasks taking place are when the written tasks such as the deliverables are under construction.  The block diagram below depicts the rough division of work between each group member.
 
 
 
[[Image:WorkloadBreakdown_blockdiagram.jpg|300px|centre]]
 
 
 
As can be seen in the Gantt Chart there are 4 major deliverables throughout the course of the project. These are:
 
*Proposal Seminar
 
*Stage 1 Design Document
 
*Final Seminar
 
*Final Report
 
 
Each of the deliverables is allocated sufficient time to be completed.  The time allocated to each deliverable has been designated with the intention of causing as little stress on other components of the project as possible.  This involves starting the construction of deliverables as soon as there has been sufficient project progress to report.
 
  
 
===Budget & Resources===
 
===Budget & Resources===
Each student is provided with a $250 budget<ref> Project Handbook 2010 [[File:Project_handbook_2010_V-20100310.pdf]]</ref>.  With 2 students involved in this project the total project budget is $500. Fortunately the progress of this project relies heavily on software and past results which are both readily accessible through pre existing (free) university resources and the internet. It is estimated that this project will fall well under budget.  With no hardware necessary, all uses for the budget will be made early on in the project life for research purposes.
 
  
Cost so far:
 
 
The Secrets of Codes: Understanding the World of Hidden Messages by Paul Lunde - $50
 
  
 
===Risk Management===
 
===Risk Management===
One great benefit of this project is that it is heavily software based.  This means that the risks and constraints associated with the project are kept to a very minimal level. With no hardware, or need for relying on external sources (as far as we are currently aware), there should be no issues with time constraints due to any third party delays. However like all projects there are risks associated with this one.  The following table depicts the possible risks as well as their relevant likelihood and importance.  Ratings are given from 1 to 10 with 10 being the highest probability/impact on the project.
 
 
 
 
<center>
 
{| class="wikitable" border="1"
 
|-
 
!  Risk
 
!  Probability
 
!  Impact
 
!  Comments
 
|-
 
|  Project member falls sick
 
|  2
 
|  9
 
|  Could leave 1 member with a huge amount of work to complete
 
|-
 
|  Unable to contact project supervisors
 
|  2
 
|  6
 
|  If it is at a critical time problems may arise
 
|-
 
|  Crawlers not available/useful in Java
 
|  4
 
|  6
 
|  New programming language may need to be learnt and implemented in very little time
 
|-
 
|  Run out of finance
 
|  1
 
|  7
 
|  Has the potential to be a problem if software needs to be purchased
 
|-
 
|  Somebody else cracks the code
 
|  .009
 
|  5
 
|  The project may lose its "x" factor but the web crawler will still be useful
 
|}
 
</center>
 
 
 
 
===Occupational Health & Safety===
 
The two main Occupational Health and Safety hazards identified are psychological and ergonomic hazards<ref>Hazards and Risk Assessment Powerpoint Presentation, Honours Project 2010</ref>. Both of these are quite likely to occur during the length of the project and can be potentially severe if not handled properly.
 
The ergonomic hazards that apply to this project include:
 
 
*Work placement layout
 
*Work place equipment
 
*Job design
 
*Individual work practices
 
*Training/Experience
 
 
The University of Adelaide has official practices for managing these ergonomic hazards<ref>Work Station Ergonomics http://74.125.95.132/search?q=cache:SofKcuyfEPYJ:www.adelaide.edu.au</ref>.  The majority of the work in this project will likely take place on computers at home or at the University, so these strategies can be implemented throughout the course of the project to minimise the risks and cause the project to run more smoothly.
 
 
Throughout the project, frequent short rest breaks should be taken. This is to help prevent fatigue and permanent muscular, joint and ligament damage. This decreases both the severity of the hazard and the likelihood that it will occur.
 
 
Additionally, the University has a Health Safety and Wellbeing Policy<ref> Health Safety and Wellbeing Policy http://www.adelaide.edu.au/hr/ohs/handbook/</ref>. Following these guidelines and principles will help avoid the psychological hazards such as:
 
 
*Stress
 
*Repetitive tasks
 
*Little training or instruction
 
 
The listed hazards are highly probably, but with proper consideration, we can minimise the consequence or hopefully prevent the hazard altogether. The high workload of the project will often lead to stress; this can be prevented by breaking the project down into smaller manageable tasks, dividing the work between members or taking short breaks during long periods of work. Due to the nature of the project, we will have to perform repetitive tasks such as testing and modifying the code and analysing similar results. We can avoid this hazard by having group members alternate between what assignment we currently have.
 
 
Finally, the project is largely independent and we have little instruction on how to complete it. This hazard is not so significant because we have two supervisors that we can keep in contact with through e-mail and regularly scheduled meetings.
 
  
With proper mitigation strategies in place, the probability of OH&S hazards occurring can be minimised. Without these management strategies in place, the potential for risk could be considered too high. However, because the risks can be effectively managed, it is reasonable to undertake the project.
 
  
 
==Conclusion==
 
==Conclusion==
  
Ideally by the end of this project the Somerton Man Code will be broken and his long lost identity will finally be revealed to the world.  This is not however the main objective of the project.  The structure of the project has been designed to produce a useful web searching tool that may be used to progress the investigation into the Taman Shud case, but will also provide uses outside of this project.  There are still some elements of the project that hold some uncertainties, however the basic outline of the project is clear and will be held to its time frame to the best of our ability.
 
  
 
==References==
 
==References==
<references/>
 
  
 
==See also==
 
==See also==

Revision as of 12:36, 25 May 2010

Due Date

edit****

Executive Summary

For decades the written code that is linked to the “Summerton Man Mystery” (or Taman Shud Case) has been undecipherable and delivered no solid information about his history or identity. This project aims to bring light to the unsolved mystery by tapping into the wealth of information available on the internet.

Recent collection of random letter samples has been used to reinforce previous claims that the code is unlikely to just be a randomly generated sequence of letters. Along with thorough revision and checking of previous code used to produce results, it has been agreed that the Summerton Man code is most likely initialism.

The code found with the dead man will be used in conjunction with a suitable “Web Crawler” and specific pattern matching algorithms. The aim is to target websites of interest and filter through the raw text available in search of character patterns that are also present in the code. Once some solid results are obtained, frequency analysis will hopefully enable a solid hypothesis on some possibilities in regards to the meaning behind the code.

unfinished***


Aims and Objectives

The aim of the project is to make progression with respect to the Summerton Man Case through use of engineering techniques such as information theory, problem solving, statistics, encryption, decryption and software coding. Through use of such techniques, the results from the previous year’s project will revised and used to solidify the pathway for the rest of the project. Once past results are properly revised the next aim of the project is to attempt to utilise a web crawling device in order to access information stored in raw text on websites of interest. Specifically designed algorithms will then be used on the sourced text in order to search for patterns and reoccurrences that may be of interest or relation to the contents of the Summerton Man Code.

Project Background

Requirements and Specifications

include block diagrams/workload breakdown

Progress So Far

Approach for remaining part of the project

Project Management

Budget & Resources

Risk Management

Conclusion

References

See also