Cracking the Voynich code 2014 weekly progress

From Derek
Revision as of 19:28, 1 November 2014 by A1210982 (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

Weekly progress and questions

This is where you record your progress and ask questions. Make sure you update this every week. The deadline is every Friday evening. However, if you sometimes slip a little into the weekend (so long as you don't do it too often) we won't be too hard on your marks.

Please remember that we make use of this progress section to give you your project mark. Your mark will suffer if you don't complete this section.

Briefly record (i) progress, (ii) problems/questions, (iii) goals for following week.

Semester 1 Week 1

Bryce

  1. Week 1 progress:
    • Group meeting with supervisors (Derek, Maryam) and project partner (Peter)
    • Started reading and discussing background information on Voynich Manuscript
    • Attended literature training session with Kay Leverett (Friday, 9am)
  2. Goals for Week 2:
    • Begin article searching and research into the Voynich Manuscript
    • Start on Project Proposal preparation

Peter

  1. Week 1 progress:
    • First group meeting with supervisors
    • Started reading articles about current progress made towards a translation
    • Found and discussed the merits of some of the available ascii representations of the manuscript
    • Attended literature search training with Kay Leverett
  2. Goals for Week 2:
    • Start on project proposal preparation
    • Write some basic scripts to help with formatting and searching the ascii files
    • Continue researching the Voynich Manuscript and the attempts that have been made to decode it


Semester 1 Week 2

Bryce

  1. Week 2 progress:
    • Attended the Risk Assessment and Management lecture (Thursday)
    • Met with Peter and discussed plans and content for the proposal seminar:
      • Organised times for which to do the presentation
      • Looked at previous years proposal presentations
    • Found previous comparisons on word frequency/occurrence and plots people have done on the Voynich Manuscript
  2. Goals for Week 3:
    • Have a draft of the proposal presentation ready for Monday
    • Finalise and practice

Peter

  1. Week 2 progress:
    • Attended Risk Management Lecture
    • Met with Bryce and discussed proposal seminar
    • Continued research on past VMS decoding work
  2. Goals for Week 3:
    • Complete Proposal Seminar Draft Slides by Monday
    • Discuss Draft Slides with Supervisor
    • Finalise and practice seminar


Semester 1 Week 3

Bryce

  1. Week 3 progress:
    • Discussion of project proposal with supervisor and team (Tuesday)
    • Added comparison table, video and WRI slides
    • Edit and finalise presentation
    • Presented at proposal seminar (Friday 4:35pm)
  2. Goals for Week 4:
    • Investigate alternate plans and suggestions
    • Further research
    • Continue as per the Gantt chart and WBS

Peter

  1. Week 3 progress:
    • Discussed Proposal Seminar Draft with Supervisor (Tuesday)
    • Added SVM, MDA, HMM theoretical slides to presentation
    • Completed and edited presentation slides and notes
    • Proposal Seminar (Friday 4:35pm)
  2. Goals for Week 4:
    • Discuss and refine project planning
    • Continue with phase 1 of project plan


Semester 1 Week 4

Bryce

  1. Week 4 progress:
    • Management aspects of project
    • Unfortunately nothing substantial as of Friday
    • Meeting on Sunday with team
  2. Goals for Week 5:
    • Supervisor and team meeting for progress report and analysis
    • Continue as per the Gantt and WBS

Peter

  1. Week 4 progress:
    • Prototype scripting with voytools and matlab/octave
    • Organised Team Dropbox to better suit project work
  2. Goals for Week 5:
    • Meet with team and supervisor
      • Update and finalise gantt chart/WBS
      • Discuss cloud storage options for code
      • Discuss current code work
    • Continue with phase 1 as per gantt chart


Semester 1 Week 5

Bryce

  1. Week 5 progress:
    • Had meeting with supervisor and team on Tuesday
      • Possible modification of analysis technique to better suit
      • Reduced scope of analysis to Takahashi only
    • Approval for purchase of resources
      • Sent to Danny (Storeman) for purchase
    • Tested new scripts in MATLAB for arranging data
  2. Goals for Week 6:
    • Finish sorting EVT file into something workable
      • Produce some outputs/comparisons
    • Informal meeting for progress update

Peter

  1. Week 5 progress:
    • Had meeting with supervisor and team (Tuesday)
  2. Goals for Week 6:
    • Finish sorting EVT file into something workable
      • Produce some outputs/comparisons
    • Informal meeting for progress update


Semester 1 Week 6

Bryce

  1. Week 6 progress:
    • Chased up order (1) with Danny
    • N/A
  2. Goals for Week 6b:
    • Progress update e-mail
    • Complete week six (6) goals this Sunday
    • Team meeting
      • Should note that this weekend is UofA MMW2, services affected

Peter

  1. Week 6 progress:
    • Experimented with matlab and python solutions to Phase 1
  2. Goals for Week 6b (Holidays)
    • Complete Phase 1


Semester 1 Break

Bryce

  1. Semester Break progress:
    • Searchable format complete, requires manual verification
  2. Goals for Week 6b:
    • Begin progress report

Peter

  1. Semester Break progress:
    • Further scripting work
  2. Goals for Week 7:
    • Finish Phase 1
    • Team meeting

Semester 1 Week 7

Bryce

  1. Week 7 progress:
    • Had team meeting to discuss progress (Wednesday)
    • Edited progress report wiki
    • Matlab script edited and simple tests verified
  2. Goals for Week 8:
    • Meeting with Supervisors
    • Continue with scripting and ranking of occurrences

Peter

  1. Week 7 progress:
    • Team Meeting
    • Further scripting work
      • Words per page
      • Most common words
  2. Goals for Week 8:
    • Meet with supervisors
    • Finish Phase 1

Semester 1 Week 8

Bryce

  1. Week 8 progress:
    • Team meeting with supervisors Derek and Maryam (Wednesday)
    • Extra ideas and different methods are being introduced
    • Matlab script added, ranking comparisons
  2. Goals for Week 9:
    • Clean up Matlab script, streamline. Implement into 6 sections
    • Begin WRI analysis

Peter

  1. Week 8 progress:
    • Team Meeting with supervisors
    • Gathered statistics about ngrams present in the VMS
    • Uploaded meeting minutes
  2. Goals for Week 9:
    • Begin Phase 2: identify unique words based on VMS Images
    • Investigate methods for normalising VMS statistics discussed in the meeting

Semester 1 Week 9

Bryce

  1. Week 9 progress:
    • Ongoing modifications to Matlab script
    • No other significant progress this week
  2. Goals for Week 10:
    • Complete WRI comparisons and other suggested ideas from meeting
    • Finish simplifying Matlab script
    • Continue with progress report update

Peter

  1. Week 9 progress:
    • Researched different metadata types available in EVT file
    • Recorded 3-4 word descriptions for all VMS images
  2. Goals for Week 10:
    • Identify unique words on pages with certain image types
    • Begin progress report

Semester 1 Week 10

Bryce

  1. Week 10 progress:
    • Planned progress report items
    • Major simplifications to code almost complete
  2. Goals for Week 11:
    • Continue simplifying Matlab script
    • Continue with progress report
    • Fix coding issues with word detections

Peter

  1. Week 10 progress:
    • Modularised Existing Code
  2. Goals for Week 11:
    • Write test cases for existing code
    • Develop method for testing the UNDHR
    • Write python script to split VMS into sections


Semester 1 Week 11

Bryce

  1. Week 11 progress:
    • Word detection fixed, tested and accurate within EVA file
    • WRI code written, issues need fixing
  2. Goals for Week 12:
    • Revise current coding attempts and print graphs
    • Continue with progress report
    • Group meeting and project discussion

Peter

  1. Week 11 progress:
    • Wrote python script to split VMS into sections based on certain qualifiers
    • Wrote python script to sort by Zandbergen illustration and find unique words
    • Tested unique word cases for various sections
  2. Goals for Week 12:
    • Write further test cases
    • Develop method for testing the UNDHR
    • Compile progress report


Semester 1 Week 12

Bryce

  1. Week 12 progress:
    • Modified coding scripts for some statistics
    • Added test functions and checking
    • Plots added to progress report and sections added and edited
    • Discussion of project progress with supervisors and team (Wednesday)
  2. Goals for Week 13:
    • Begin editing code into functions
    • Read more into the books ordered

Peter

  1. Week 12 progress:
    • Wrote test cases for the UNDHR and for various other English texts
    • Tested current scripts and compiled results
    • Wrote and formatted progress report wiki
  2. Goals for Week 13:
    • Expand on Phase 2 (Picture Identification) as explained in the progress report
    • Read "An elegant enigma"


Semester 1 Week 13

Bryce

  1. Week 13 progress:
    • Week been dedicated to exam revision
    • Discussed ideas and project direction
  2. Goals for Week 14 (SWOTVAC):
    • Finish editing code into functions
    • Begin on testing subsets of manuscript including WRI expansion

Peter

  1. Week 13 progress:
    • Read about half of "An elegant enigma"
    • NOTE: Most of week dedicated to study for other subjects
  2. Goals for Week 14 (SWOTVAC):
    • Finish NSA Study
    • Expand on Phase 2 (Picture Identification) as explained in the progress report


Semester 1 Week 14 (SWOTVAC)

Bryce

  1. Week 14 progress:
    • Finished reading 'An Elegant Enigma'
    • Continued as per usual
  2. Goals for Semester Break:
    • Look at other similar texts to use for comparison purposes and to reduce text size for future tests

Peter

  1. Week 14 progress:
    • Finished reading "An Elegant Enigma"
    • Developed code for sectioning out a certain subset of pages easily
  2. Goals for Semester Break:
    • Try to get some Voynich work done while in Chile

Semester 2 Week 1

Bryce

  1. Week 1 progress:
    • Transcribed a herbal text dating back to the mid 1500s for comparison, medieval Gothic
    • Started writing methods for foreign text determinations
    • Researched and read methods of literature comparisons plus findings
  2. Goals for Week 2:
    • Have a group meeting with Peter discuss methods and ideas and progress
    • Fix word finding algorithm for faster run time and accuracy

Peter

  1. Week 1 progress:
    • Designed an algorithm for testing picture descriptors
    • Read Zanbergen and Stolfi's previous work on the manuscript grammar and word entropy
    • Group meeting with Bryce
  2. Goals for Week 2:
    • Continue reading Stolfi's thoughts on grammar and language comparisons
    • Test word diversity code on other standard-length texts

Semester 2 Week 2

Bryce

  1. Week 2 progress:
    • Fixed word finding abilities within code for accuracy
    • Implemented several statistical methods mentioned earlier, basic statistics
    • Reduced statistical comparisons to only Herbal A and Herbal B languages
    • Group meeting and project discussion with Peter
  2. Goals for Week 3:
    • Modularise code fragments and test cases for robustness and accuracy
    • Begin testing sample texts and outputs and write up results

Peter

  1. Week 2 progress:
    • Read about possible coarse and fine grammar structures for the VMS
    • Tested word diversity and alphabet length for a number of languages, including the 1500s herbal text
  2. Goals for Week 3:
    • Rewrite code for Unicode language support
    • Look at basic Voynich grammar indicators

Semester 2 Week 3

Bryce

  1. Week 3 progress:
    • Completed algorithm simplification for faster and easier output comparisons
    • Tested scripts using 2 extra sample data sets both transcribed texts
    • Started WRI in conjunction with SVM foundations for data testing in next phase
  2. Goals for Week 4:
    • Finish off WRI modifications, begin SVM algorithm and classification method
    • Supervisor meeting with project partner Peter for discussion and updates
    • Complete write up for foreign text determinations

Peter

  1. Week 3 progress:
    • Rewrote code to support Russian, Hebrew, Hungarian, and Chinese
  2. Goals for Week 4:
    • Fix Chinese punctuation errors
    • Meet with Bryce

Semester 2 Week 4

Bryce

  1. Week 4 progress:
    • Informal project meeting with Peter
    • More SVM modifications required, begin testing with larger data sets and languages
  2. Goals for Week 5:
    • Organise meeting for project update
    • Improve language detection for code

Peter

  1. Week 4 progress:
    • Researched algorithms for fuzzy string matching (edit-distance)
    • Wrote code to look at which words are most likely to be "alternate spellings" based on edit distance and heuristics
    • Group meeting with Bryce
  2. Goals for Week 5:
    • Meet with supervisors
    • Basic Voynich grammar indicators
    • Improve heuristics by testing "mispelling" code against old-english texts

Semester 2 Week 5

Bryce

  1. Week 5 progress:
    • Project meeting with supervisors (Friday)
    • Looked into Word Entropy and information for words in Voynich
  2. Goals for Week 6:
    • Complete entropy for words within the Voynich Manuscript
    • Continue to optimise code functions for efficiency for statistics

Peter

  1. Week 5 progress:
    • Met with supervisors (Derek) (Friday)
    • Read Zanbergen tests on language clouds
    • Started designing new experiments
    • Wrote proximity letter search script to identify ngrams
    • Wrote script to identify word position probability for each EVA character
  2. Goals for Week 6:
    • Match phrases in the VMS to identify alternate 'word' spellings
    • Extend grammar identification scripts
    • Develop experiments based on Zandbergen notes
    • Extend testing of grammar scripts to other languages.

Semester 2 Week 6

Bryce

  1. Week 6 progress:
    • Used different parts of the Voynich A and B languages with texts from languages within the period for analysis
    • Examined authors and texts within the period to find similar comparisons for later use
  2. Goals for Week 7:
    • Complete word entropy analysis
    • Researched into Matlab functions for language statistics and methods
    • Experiment with other software for analysing and presenting

Peter

  1. Week 6 progress:
    • Researched the Longest Repeated Substring Problem
    • Researched suffix trees as an O(n) solution to long phrase matching with edit distance metrics
    • Failed to find appropriate suffix tree library for python (C++ is the main language used for string match algorithms)
    • Revised problem scope after identifying the lack of long (greater than 4 words) repeated phrases in the VMS
  2. Goals for Week 7:
    • Rewrite code to fix several issues with unicode support and modularity
    • Find simple solution to the phrase matching problem

Semester 2 Week 7

Bryce

  1. Week 7 progress:
    • Discussion of word statistics and methodology with Peter
    • Continue with WRI with smaller sets of data and accuracy
    • Fixed coding issues within functions
  2. Goals for Week 8:
    • Finish off statistical analysis
    • Begin into SVM training of data and data groupings

Peter

  1. Week 7 progress:
    • Identified a number of 2-4 word repeated phrases in the VMS
    • Rewrote existing code for ease of readability, better modularity, better CJK language support, and python3 compatibility
    • Researched collocations (n-grams), word relationship metrics, and n-gram language models
    • Did an experiment on the number and ranking of collocations in the VMS versus representative texts for several known languages
  2. Goals for Week 8:
    • Write code to match desired n word phrases with other close phrases
    • Find hand spaced Chinese and/or Japanese texts for comparison purposes

Semester 2 Week 8

Bryce

  1. Week 8 progress:
    • Completed test cases for word length distributions on complete UDHR
    • Finished off data analysis on WRI for test cases on different languages
    • Research into better SVM methods and MDA analysis
  2. Goals for Week 8a:
    • Project meeting with supervisors and discussion about methods
    • Completed modifications to allow language comparisons within Matlab

Peter

  1. Week 8 progress:
    • Developed and tested phrase matching code
    • Reread Knight study and other VMS theories for ideas on possible experiments
  2. Goals for Week 8a:
    • Research possible uses of two-state HMMs
    • Test inclusion of ambiguous spaces (current tests have removed ambiguous spaces from the EVT)
    • '-am' character groups may signify an end-of-line punctuation character (Nick Pelling)
    • Look into possible simplified alphabets: find key letter groupings of n EVA characters which may be 1 true character

Semester 2 Week 8a (Semester Break Wk 1)

Bryce

  1. Week 8a progress:
    • Project meeting with supervisors (Wednesday)
    • Meeting with Maryam to discuss MDA and SVM pathway
    • Fixed word length distributions
    • Basic SVM analysis for 2 groups of data
    • Optimised code to function quicker
  2. Goals for Week 8b:
    • Continue with SVM, analyse larger distributions of data with more groups
    • Extra analysis on the UDHR with WRI

Peter

  1. Week 8a progress:
    • Met with supervisors (Derek and Maryam) (Wednesday)
    • Wrote code to identify possible character groups that only appear at the end of a word or line in the VMS
    • Analysed the possibility of punctuation or cipher characteristics in the VMS
  2. Goals for Week 8b:
    • Test ambiguous space inclusion and isolation of small contiguous segments of the VMS
    • Develop a way to show character relationships easily in the final report

Semester 2 Week 8b (Semester Break Wk 2)

Bryce

  1. Week 8b progress:
    • Read up on statistical analysis of languages and methods
    • SVM successfully analysing groups of data, categorisation fixed
    • Continue work on WRI and UDHR data
  2. Goals for Week 9:
    • Complete MDA analysis
    • Finish final seminar preparation and slides
    • SVM analysis complete and output write up

Peter

  1. Week 8b progress:
    • Looked at letter n-grams as a way to show character relationships in the VMS and comparison texts
    • Began final report writing process
    • Finished final list of main comparison texts
    • Looked at Shannon Entropy of the VMS vs. comparison texts
  2. Goals for Week 9:
    • Develop Formal transcription testing method to help future research.
    • Hidden Markov Model development
    • Extend entropy code to bigrams and trigrams

Semester 2 Week 9

MONDAY 6th OCT (PUBLIC HOLIDAY)

Bryce

  1. Week 9 progress:
    • Project meeting with supervisors and discussion (Tuesday)
    • Final project preparation, lamination and printing of complete book done
  2. Goals for Week 10:
    • Complete the final seminar project items required
    • Complete project final close off and assessment items

Peter

  1. Week 9 progress:
    • Meeting with project supervisors (Derek and Maryam) on Tuesday
    • Finished code which shows transcriptions completeness, approximate accuracy, and the reliability of each section of the manuscript
    • Turned UDHR into a testable document for WRI and word length distribution code
    • Added Pinyin Chinese text to main comparison text list.
    • Printed VMS reference text.
  2. Goals for Week 10:
    • Add bigram and trigram to entropy comparisons
    • Compile analysis and graphs for report/seminar

Semester 2 Week 10

Bryce

  1. Week 10 progress:
    • Completed MDA and SVM comparisons with language texts
    • Completed final seminar slides and notes
    • Completed UDHR testing and files for Matlab
    • Final preparations for expo and project write up
    • Final Seminar: Thursday, 16th October 2014, 3pm
  2. Goals for Week 11:
    • Complete final report and poster for expo
    • Finish lamination and binding of documents
    • Complete YouTube video of project

Peter

  1. Week 10 progress:
    • Some code redesigns to make results easier to understand
    • Completed files for python
    • Wrote results sections of Final Report
    • Completed final seminar slides and notes
    • Final preparations for expo and project write up
    • Final Seminar: Thursday, 16th October 2014, 3pm
  2. Goals for Week 11:
    • Complete final report and poster for expo
    • Finish lamination and binding of documents
    • Complete YouTube video of project

Semester 2 Week 11

Bryce

  1. Week 11 progress:
    • Completed final project report
    • Final preparations for expo and YouTube video
  2. Goals for Week 12:
    • Complete the exhibition poster
    • Finish editing of the YouTube video
    • Attend exhibition and prepare exhibition materials

Peter

  1. Week 11 progress:
    • Completed final project report
    • Final preparations for expo and YouTube video
  2. Goals for Week 12:
    • Complete the exhibition poster
    • Finish editing and voiceover of the YouTube video
    • Attend exhibition and prepare exhibition materials

Semester 2 Week 12

Bryce

  1. Week 12 progress:
    • YouTube video preparations
    • Project poster completed
    • Exhibition of project at Convention Centre (30th October)
  2. Project Closeout:
    • Finalise the YouTube video
    • Save project directories to USB/CD
    • Final hand in

Peter

  1. Week 12 progress:
    • Finished video design for Youtube
    • Project poster completed
    • Bound Reference Copy of Voynich
    • Exhibition of project at Convention Centre (30th October)
  2. Project Closeout:
    • Complete voiceover for Youtube
    • Save project directories to USB/CD
    • Hand in code and sign off on deliverables

END OF PROJECT

See also


Back

Retrieved from "https://www1.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php?title=Cracking_the_Voynich_code_2014_weekly_progress&oldid=14863"