War Diaries Talk

Name Data from Operation War Diary

  • milh0use by milh0use

    Hi All,

    My name is Steven Hirschorn and I'm on the National Archives team for the Operation War Diary project. I've been working the past two years on the data that has been coming from our volunteers, attempting to find the best ways of clustering the tags from different users and establishing the consensus viewpoint of what is on the page.

    There have been many challenges along the way, which some volunteers may have heard about in the Catalogue Day talk I gave relating my experiences working with the data. I'm quite happy with the algorithms that do the clustering and consensus now, and have uploaded data by diary to Google Drive. Also included is the dataset I supplied to Professor Richard Grayson which aided his research for his journal article, mentioned previously in this forum.

    One of the datasets that I think is of immediate value without any interpretation or analysis is the name data from the diaries. These names are listed in the by-diary datasets, but I thought I'd also create a name index so that anyone looking for a particular person can do an alphabetical lookup. I've created a Tab-delimited text version (for processing with a spreadsheet application or a programming language) and an HTML (web page) version. Both have the same list of names, ranks, units, and pages that the person is mentioned on. The HTML version also has a hyperlink to Discovery, the National Archives Catalogue, from where you can download the original diary to read further. It also has a link to the search engine in Imperial War Museums' Lives of the First World War website, where you may find extra information. Please note that you will need to download the HTML page before viewing it, as Google Drive defaults to displaying the HTML Source Code. After you've downloaded the file it should open up in your favourite web browser. The file is 32MB, so may take a minute or two to download depending on your bandwidth.

    The list contains around 74,000 rows. There are errors, often mis-spellings by the adjutant who wrote the diary, so the same person may have multiple entries, with slight spelling variations. Also there may be references to several different people in a single row, because it is not possible to say whether or not "Private Smith" mentioned on page 7 of a diary is the same person as "Private Smith" on page 35 or "Private T Smith" on page 70. These are the same challenges that anyone reading the diary without the Operation War Diary data would need to consider.

    I hope the dataset is useful, please let me know if you have any problems downloading it or accessing it, or if it would be useful to you in a different format. Likewise, if you have any ideas of things you'd like to do with the data, I may be able to help. And last but not least, thanks for your efforts to date in helping generate a structured, indexable dataset from the diaries.

    Here's the folder with the Tab-delimited text file and HTML file:
    https://drive.google.com/open?id=0BxfXwWCjrmUMamZ5RTVyQVpXcWM

    Regards,
    Steven

    Posted

  • ral104 by ral104 moderator, scientist

    This is fantastic - thanks for making it available!

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Thank you for the time and effort you have taken to format this information for us, no problem downloading this data and I have opened the txt file in Excel, which formats it nicely. Very handy indeed! Nice to know how the names we are tagging are processed behind the scenes. Another addition to my WW1 folder. Thank-you again!

    Posted

  • milh0use by milh0use

    I've just realised I forgot one of the two links I was going to post! Here's the link to the datasets for each diary. The exports are all tab delimited text (for loading into excel or processing with a programming language) and include the person names and place names tagged from the diaries. For the moment I've left out the georeferences (latlongs) for the places because they do need a bit more work.

    The folder below also has the datasets I sent to Professor Grayson, with activity counts per day.

    Again, please do let me know if you have ideas for things you'd like to do with the data, particularly if it would help to have the data formatted differently.

    https://drive.google.com/open?id=0BxfXwWCjrmUMfm1oQU9zOFJvbEkwdzQxXzBZdXVPRUhHaEJIWk9FRHdqdjlBbVRpSzRjOTg

    Posted

  • RobertAdamson by RobertAdamson

    Hello Steven
    Thank you for all your work & letting me be part of it.

    Stupid question, how can I search your data?
    Also how can I check whether a specific diary is available?

    I'm looking for anything about my grandfather JH Lane 2/4 bin KOYLI who died around 20 March 1918 at the start of the German offensive Operation Michael

    Thanks

    Robert

    Posted

  • cyngast by cyngast moderator in response to RobertAdamson's comment.

    Hi, Robert,

    In answer to your question, I'm not sure how you can search all of Steven's data other than to use the links he has posted here.

    I can tell you, however, that the diary for your grandfather's unit, the 2/4 Battalion KOYLI, which was part of the 62nd Division, has not yet been tagged, so the National Archives database for Operation War Diary would not have anything on your grandfather. You can download the battalion's diary through the NA for a fee, but you'll have to check with the NA as to the actual amount.

    I've just run a search on the Commonwealth War Graves Commission website and found your grandfather. His information includes his name, service number and bit of family information.

    Also, Steven does not really use the Talk boards here. I don't know if you can reach him directly at the NA.

    If you have any further questions, please ask. We're always happy to help.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Hello Robert, I looked at the section on Steve's data- (Global Date index )
    https://drive.google.com/open?id=0BxfXwWCjrmUMamZ5RTVyQVpXcWM

    I entered the name Lane but it did not appear.

    I did this by going into the link, Pressed the Control Key and the F key (find). At the top of the page there will appear in the top right hand corner a small square box – just type in the name Lane and if it is there it will come up in colour, which is easy to find, it will show also how many times the name appears in the document if it is there – you probably know this already.

    You can also search the (Global Name index) by using the same method, i.e. Cavalry Division – just type in Cavalry Division in the small box top right and press enter - any mention of the Cavalry Division will come up in colour.

    Memorial Information for the KOYLI ww1: - KINGS OWN YORKSHIRE LIGHT INFANTRY REGIMENTAL CHAPEL. YORK MINSTER,Deangate,York, North Yorkshire, England, OS Grid Ref.: SE 60331 52179, Denomination: Church of England Link: http://www.iwm.org.uk/memorials/item/memorial/30490

    Information for you on The Commenwealth War Graves Commission: Link: http://www.cwgc.org/find-war-dead.aspx?cpage=1

    Giving: Name/Rank/Service No/Date of Death/Age/Regiment/Grave Memorial Ref/Cemetery Memorial Name. I hope this is the person you were looking for.

    Posted

  • RobertAdamson by RobertAdamson

    Thanks, that was kind of both of you.
    Cyngast I was interested that the relevant war diaries haven't been processed yet, I suppose patience is a virtue!

    I already have the relevant CWGC entry & some other info about enlistment place/date from a war memorial site soI guess I just need to wait

    Posted

  • cyngast by cyngast moderator in response to RobertAdamson's comment.

    I wish I could tell you when your grandfather's battalion's diary will be up for tagging here, but I don't know. The selection of which diaries will be loaded in future batches is determined by the NA.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Hello Robert, I forgot to mention that I tagged the Diary in January this year, for the 2/4 Battalion KOYLI (17 pages only) but it ran from 1 March 1919-31 Aug 1919. When they went to Gymnich, Germany

    Posted

  • Lorraine8 by Lorraine8

    Hello

    I have just signed up to help with this tagging project but the War Diary that I would be first interested in tagging does not seem to be here - 7th Battalion, Rifle Brigade. Please could you let me know when I might be able to tag this War Diary?

    Thank you

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Hello Lorraine, Welcome to the project you will find everyone on the project very helpful.

    Here is some information for you about the 7th Battalion Rifle Brigade. I don't think this diary has been up. The Moderator would know. I believe they were part of the 41st Brigade, 14th Light Division. See link: https://www.forces-war-records.co.uk/units/1508/kings-royal-rifle-corps/ Here is another link re 7th Btn Rifle Brigade. http://www.wartimememoriesproject.com/greatwar/allied/alliedarmy-view.php?pid=6868

    Your question really interested me, because I have been meaning to post a question about the Rifle Brigade for a long time. My grandfather was part of the Rifle Brigade and came back wounded in 1916. It appears from my paper work regarding him (of which I have given you the links) that he may have been part of the 7th or 8th Btn of the Rifle Brigade – 41st Brigade 14th Light Division. They seem to have been part of the same battles.

    I am going to ask the Moderator some questions about the Rifle Brigade which I want to query. I hope to post this after have posted this to you.

    Posted

  • cyngast by cyngast moderator

    Hi, Lorraine and Welcome!

    To both you and Marie, we have not yet had any diaries from the 14th Division up for tagging. I can't tell you when we will, as the National Archives controls the schedule for uploading diaries. We are currently waiting for a new batch, but we don't know exactly when it will turn up or what it will contain.

    Marie, as you asked in the other thread, all diaries that have been digitized can be downloaded from the NA, and we have been told they are all digitized now. There is a fee, but I don't know what it is. The NA website is not specific about the amount either, so it may depend on the length of the diary, but that is just a guess on my part.

    Lorraine, Please feel free to ask any questions you may have. Tagging through these diaries seems to raise all kinds of questions from technical issues to what did the author mean to general history of the war. I've been working on them for three years now and I still have questions! This is a friendly forum and everyone is willing to help out.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Cynthia, have you got a link for downloading diaries from the National Archives? I am sure someone mentioned it but can't remember when. I think the fee is quite small.

    Posted

  • cyngast by cyngast moderator

    I think this is the best place to start: http://www.nationalarchives.gov.uk/help-with-your-research/research-guides/british-army-war-diaries-1914-1922/ This page explains how to search for a specific regiment, but if you put in a battalion number, such as 9th Rifle Brigade, it says it can't find any results. Rifle Brigade brings up a list of all the units in the regiment, though.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Cynthia, thanks for that link for ordering diaries to tag.

    Link: for ordering http://discovery.nationalarchives.gov.uk/details/r/C7352730

    The 7th and 8th Btns Rifle Brigades are on this link: http://discovery.nationalarchives.gov.uk/results/r/2?_q=Kings Royal Rifle Brigade&_col=200&_cr1=WO 95&_hb=tna

    Will post this for Lorraine who was interested in the 7th Battalion Rifle Brigade.

    Over 5 years ago when I lived in Plymouth, South Devon, I telephoned the National Archives at Kew regarding my research about my Grandfather in the Rifle Brigade, naming the 7/8/9th Btns of the Rifle Brigade. I think they gave me the incorrect information. I was told that there were no diaries for the 7th/8th/9th Battalions of the Rifle Brigade.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Message for Lorraine re 7th Battalion Rifle Brigade. You may be interested in these links.

    Link: for ordering http://discovery.nationalarchives.gov.uk/details/r/C7352730

    The 7th and 8th Btns Rifle Brigades are on this link: http://discovery.nationalarchives.gov.uk/results/r/2?_q=Kings Royal Rifle Brigade&_col=200&_cr1=WO 95&_hb=tna

    Posted

  • ral104 by ral104 moderator, scientist

    Just to add to this - the Rifle Brigade and the King's Royal Rifle Corps were separate regiments during the Great War. So searching for something like '7 Rifle Brigade' within the WO95 record set might bring back fewer results. Easier to sort through 😃

    http://discovery.nationalarchives.gov.uk/details/r/C7352734 is the 7th Bn Rifle Brigade.

    Posted

  • cyngast by cyngast moderator in response to marie.eklidvirginmedia.com's comment.

    Marie, I just noticed you thanked me for the information for ordering "diaries to tag."

    I just want to make sure you know that if you download diaries from the NA, you can't tag them. You just get a digitized version of the diary to read.

    Diaries have to be uploaded to our project in order to be tagged.

    Posted

  • David_Underdown by David_Underdown moderator

    The key thing when searching on Discovery is to use cardinal numbers, not ordinal as you might expect, that is search for "1 battalion", "7 battalion" etc, not "1st battalion" "2nd battalion" etc.

    so the best search is something like WO 95 "7 battalion" "rifle brigade"
    put the double quotes as I have done as then it treats those elements as an exact phrase

    Posted

  • cyngast by cyngast moderator

    Thanks for the tip, David!

    Posted