War Diaries Talk

The Future of OWD

  • ral104 by ral104 moderator, scientist

    Firstly, Happy New Year to everybody, and thanks for all the work you've put in on OWD over 2018. Together, we've completed 156,640 diary pages, which equates to 859 completed diaries.

    The project's been running for five years now, and I think it's fair to say that our main focus during that time has been tagging, using the brilliant interface developed by Jim O'Donnell. We've seen one academic paper published based on OWD's data, and Steven Hirschorn at the National Archives has done excellent work in exploring how the data can be used to build up indexes, along with ways in which we might visualise the information contained in the diary pages.

    However, things are changing. Zooniverse have developed new software to support all their projects, and will no longer be supporting the infrastructure that OWD runs on. This presents us with a significant challenge: in order to keep OWD running on the new infrastructure, we need to build a new interface. Zooniverse have done a lot of work to make this process an easy one, but so far I'm not certain it will be possible for us to recreate the rich functionality we currently have.

    This raises a number of questions, the key one being do we need to do things exactly the same way? It might be possible to continue generating the same rich, useful data we have been, while simplifying how we do it. In another thread Jim asked whether we still need to collect location data for place tags, and it's possible that we don't. We may only need to collect place names from now on, and simply match them to location data we've already collected. We may not. I'm not sure yet, but it's a good example of a way in which I don't think the new software will support us, but also how it may not matter. There's a lot to think about.

    It's also worth saying that OWD has been running as a predominantly volunteer-led project for the last several years. Like many museums, IWM was hit hard by funding cuts and their involvement fizzled out in the aftermath of that. We get a lot of excellent support from Steven Hirschorn and David Underdown at the National Archives, but there are also issues there in how compatible the data is with TNA's current systems, as well as the resource necessary to overcome those difficulties. On the community side of things, Cynthia puts an enormous amount of work into the forums and without her they would be considerably less useful. But the basic facts are that any major changes to the project will happen slowly, because none of us are working on this as our main job.

    My own opinion is that as well as continuing to generate data, we need to put more effort into using it as well. I think that's the best way of renewing interest in the project, as well as demonstrating exactly how useful it is.

    Anyway, all of this is a long-winded way of saying that the current batch of diaries will be the last on the current interface. That decision's been made for us, and there's not really much we can do to change it. I think it's very likely that the project will have some kind of hiatus while we work out exactly what to do next and how to do it. For that, I can only apologise. The effort you all put into this work is hugely appreciated, and in moving forward in a sensible way and starting to work with the data you've all generated a lot more, I hope we will make best use of everything you've done.

    I'll post more updates as things become clearer, and I'd also encourage you all to join in this discussion and let us know your thoughts on the project and any ideas you have about it.

    Thanks.

    Rob

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Rob, after reading this, is there a possibility that the project of tagging the war diaries will come to an end. I sincerely hope that is not the case after working on them for such a long time. I think this is a brilliant project and such a learning curve.

    Posted

  • ral104 by ral104 moderator, scientist

    That is one possibility, Marie, but not the one we want to happen. I feel strongly that finishing the job before it's completed would really not be fair to everybody who has put so much effort into it already.

    What I think will happen, though, is that we will have to pause the project between finishing this batch of diaries and getting the new interface in place so we can continue tagging. What I'm not sure about is how long that pause will be, but I should have a better idea once we've done more thinking about exactly what we'll need from the new interface, i.e. does it need to be the same as the current one, or can we simplify it in places?

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Thank you Rob for that quick answer.

    I would add that another good thing about this project, is the interaction between people involved, i.e. on the talk pages and discussion boards which makes you feel part of a team. So in some way I hope the project may be able to continue. πŸ˜ƒ

    Posted

  • ral104 by ral104 moderator, scientist

    I do as well, Marie. And I'm glad you've found it such a positive experience so far!

    Posted

  • deehar by deehar

    Ah well ! All good things have to come to an end ! I must say that migrating to a new platform has always filled me with dread. Of course it will be possible to tag in a different way but a lot of momentum could be lost and the fire may go out unless there are lots of interesting diaries still to be done.

    Posted

  • 141Dial34 by 141Dial34 in response to ral104's comment.

    I have finished this lot of Diaries so wow a forced hiatus! I have 59,037 pages completed so is that 159k figure correct? I have so enjoyed this project that has taught me so much about WW1 & I hope it comes back .Thank you for all the questions answered & help over the years.I am glad to have contributed.
    Regards, Sandra

    Posted

  • cyngast by cyngast moderator in response to deehar's comment.

    There are still lots of diaries to be done. Just look at this list of divisions involved in the fighting on the Western Front: http://www.longlongtrail.co.uk/army/order-of-battle-of-divisions/ There are many more divisions left than those we have completed so far. I know that some of the divisions involved in intense fighting on the Somme have not been touched yet.

    Our hopes for a new interface would be to retain the positive qualities of working through the diaries, while trying to eliminate those factors that create confusion and therefore don't produce uniform data.

    Posted

  • ral104 by ral104 moderator, scientist in response to 141Dial34's comment.

    The 159k figure is an aggregate of lots of peoples' totals - each page is tagged by at least 5 people, so it will always seem small compared to high individual page totals.

    Posted

  • deehar by deehar in response to cyngast's comment.

    Talking about factors that create confusion.... It would be nice to know where the non-uniformities occur. I often find myself choosing "clever" solutions when the options are not clear and realizing that this may not be what most taggers would do! A regular problem for me is how to tag sub-units of the unit writing the diary and their separate locations and activities. There is no feedback to taggers to stimulate convergence either.

    Posted

  • ral104 by ral104 moderator, scientist

    Actually, that's one of the things I'm interested in looking at in the existing data - how 'reliability' varies across different tags.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Ral, you mention that OWD has been running as a predominantly volunteer-led project for the last several years. Like many museums, IWM was hit hard by funding cuts and their involvement fizzled out in the aftermath of that.

    If it is a question of funding difficulties, I don't suppose the The National Lotteries could give funding for this project to continue, if a grant from them was applied for, to help commemorate and remember those who took part in WW1.

    Or would this suggestion not help at all?

    Posted

  • cyngast by cyngast moderator in response to deehar's comment.

    One specific tag that I think was not well-chosen is Withdrawing. In the Field Guide, it states to use it for troop movements in regard to consolidating the line or retreating, but in several diaries I've tagged it was used routinely when a battalion or artillery batteries were being relieved. So it might read "withdrew to billets in ABC village" instead of the more common "marched into billets in ABC village."

    I expect the data will be inconsistent for those situations where some taggers will have used Withdrawing because the diary said withdrew and other taggers will have inferred that the movement the author meant was the usual march to billets after being relieved and will have used the Unit movement tag.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    I have used the tag withdrawing when retreating and then followed it sometimes with the tag, movement if indicating they have moved to another place, i.e sometimes they have withdrawn to another place in the line and sometimes to accommodation or to another area.

    By the way do we need all the various tags for horses?

    Posted

  • cyngast by cyngast moderator in response to marie.eklidvirginmedia.com's comment.

    Marie, The day-to-day operation of OWD doesn't actually cost us anything. Zooniverse offers their services free of charge. I believe that at the time OWD was developed and launched, Zooniverse was more involved with the development of their projects but they have since moved to a more do-it-yourself type of development for new projects. I think what Rob is most worried about is that because this new interface has to be more simple and straightforward, so that people who are not extensively versed in website development can use it, we won't be able to replicate the complex system we have now. And it will take some time to sort it all out, not just because it is a big project, but because we will have to do it in our spare time.

    Rob has mentioned looking into grants in connection with looking for more ways to use the data we've all been generating for five years now. I'm not familiar with grants in the UK, but if they are similar to those in the US, the application process can be extremely time-consuming. I know of people who specialize in just writing grant proposals and make a good living doing so!

    Posted

  • cyngast by cyngast moderator in response to marie.eklidvirginmedia.com's comment.

    Marie, None of the hashtags we use, such as all the ones for horses, go into the data that the project actually generates. They are just an indication that information of a certain type, such as information about horses casualties, is present on that page. That's why we really only need for any one hashtag to appear only once in all the comments for a single page. So if the first person to tag a page adds #militarycross other taggers who follow don't need to add that hashtag in their own comment.

    Posted

  • sdorman120 by sdorman120

    Although I have no knowledge of what the technical requirements might be for introducing a new interface, I would suggest that that is less a problem than in ensuring data from a new interface and data from the old interface are compatible. If the two sets of tagged data end up being formulated differently then it may be difficult to work on aggregated data from the old and new sets. There are too many instances of government departments, commercial concerns and other organisations being unable to integrate before and after datasets after the introduction of new computer systems.

    On the wider issue of the continuance of OWD I think it would be a great loss if the work simply had to fold for lack of relevant support, but as has been mentioned earlier in this thread finding ways to actually use the data which has been produced so far could be key in ensuring that doesn't happen.

    However, another issue which may have a bearing is 'commemoration anniversary fatigue'. Obviously OWD was established in the run-up to the 100th anniversary of the start of the war and now, over four years later, we are now past the 100th anniversary of the end of the war. While people's attendance and regard for the various anniversary commemoration events which have taken place over the last few years has been tremendous, whether that can be maintained in the coming years remains to be seen and that may have a bearing on the overall future of OWD. Personally, I still consider this project to be a very worthwhile endeavour and will continue to work on it as long as possible.

    Posted

  • ral104 by ral104 moderator, scientist in response to sdorman120's comment.

    That's a good point, but actually I'm not too concerned about it - we output data in a simple format (.csv), which makes it easy to work with in a variety of ways, and we're not proposing to generate different data sets within diaries. There will be a cut-off point, and if anything changes after that, it's likely to be that we collect slightly less data. Of course, that will mean that we couldn't analyse things we dropped across the full corpus of diaries, but given that the reason we'd be likely to drop something is because it's either not useful or not well defined, I don't see that as too much of an issue.

    I would be less keen to change the type of data we collect for particular tags - that would create a disconnect between the data sets - but for now I can't think of any particular reason we'd want to do that.

    Your point about commemoration burnout is also a good one, although my sense is that by now many of our volunteers have been working on the project for a long time and do it because they're interested in the subject matter, and see it as a useful exercise. The challenge of course will be to keep that enthusiasm going even if we need to pause things for a while.

    Slightly related is Marie's question about lottery funding. The heritage lottery funding supported a lot of the centenary activities, and anecdotally at least I hear now wouldn't be the best time to apply for more funding for WW1 projects.

    Posted

  • erik.schaubroeckscarlet.be by erik.schaubroeckscarlet.be

    When (if) the project should stop temporarily in what way the taggers will be notified when it restarts?

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    That is a good question Erik, I was wondering the same. Although we will be probably be updated about the situation. I wonder if they could email us if it restarts, also how to obtain access. Glad you submitted that question πŸ˜ƒ

    Posted

  • ral104 by ral104 moderator, scientist

    My plan was to ask Zooniverse to send out an email to all registered taggers. We'd probably also try and update the facebook page, assuming we still have access to it (I haven't checked for a while).

    Posted

  • 141Dial34 by 141Dial34 in response to ral104's comment.

    Thanks Rob, I would come back in an instant so if you cant get our emails I can message it. Good Luck with funding! -Sandra

    Posted

  • ral104 by ral104 moderator, scientist

    Thanks, Sandra πŸ˜ƒ

    Posted

  • josiepegg by josiepegg

    I think we need to consider who will be using this immense database that we are creating and what information would be valuable to them? Hopefully it is not just for academic research. Actual mentions of names would seem to be of importance. There must be a lot of people who would love to be able to search a database for names of their ancestors. We have just spent 5 years doing the 800+ diaries but I am not aware of any of this information being made available to the public. Would it be possible to get something searchable online to keep things in the public eye? (For example the CLIP crew list project has an online database)

    Posted

  • cyngast by cyngast moderator in response to ral104's comment.

    Zooniverse has a system in place to alert their users when new projects are opening up. I frequently get an email saying "we need your help on a new project..."

    Posted

  • deehar by deehar in response to josiepegg's comment.

    It should be a goldmine for the genealogy market, just like the indexes created for the UK Census returns. Users could type in names and find out exactly which page of which diary to download. At present you have buy downloads of diaries and hope the person you are looking for is mentioned. The comment that worried me is that this database is not compatible with TNA software.

    Posted

  • stoat11 by stoat11

    Hi,
    I agree with all the comments here. I have been involved from pretty much the start and think it would be a very big shame if it were to end. It has been a pleasure to help.

    Mike

    Posted

  • ral104 by ral104 moderator, scientist

    Just a quick update to say I'm making some progress in setting up a new interface. It's still very early days and there are some significant issues I need to get to grips with, but it's progress of a sort. No guarantees, but so far I'm hopeful.

    Rob

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Thank you for your update for us regarding your progress in setting up a new interface, which is appreciated. Although you say no guarantees, a hopeful sign at least for us on the 'Home Front'. πŸ˜ƒ

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Ral, As I am now about to tag probably the last diary in this project, 3 left to tag. One I have tagged already. 1 left at 86% believe 2 people still tagging this. This leaves One at 60% which I shall continue to tag.

    I was just wondering if you have any more information regarding the WW1 diaries regarding a new interface.

    You said "your plan was to ask Zooniverse to send out an email to all registered taggers. We'd probably also try and update the facebook page, assuming we still have access to it"

    I never use facebook. I see Zooniverse has taken off the War Diaries from their projects.

    Therefore, I was just wondering If they will send us a message by email letting us know if the project is to be continued and how to access it.

    Would really like to continue this WW1 project if possible. I know you are working hard on the new interface. πŸ˜ƒ

    Posted

  • stoat11 by stoat11

    Hi,
    There's a twitter feed still out there I think. Last post was 9th April 2017. Might be a way of keeping us up to date with progress.

    Mike

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Unfortunately, I do not use Twitter. Thanks for your reply stoat.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Just finished the remaining pages of the last diary posted for tagging. It has been such a good worthwhile, interesting project to work on. I hope perhaps it can carry on with a new interface. I imagine also all the other people who have been involved will be of the same opinion. Hope also someone will contact us if it continues. πŸ˜ƒ

    Posted

  • deehar by deehar in response to marie.eklidvirginmedia.com's comment.

    Just finished my last diary too. Is this how it ends?

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Deehar - I was just reading Ral’s message again which he posted 2 months ago about the future of OPW .

    Ral said: β€œWhat I think will happen, though, is that we will have to pause the project between finishing this batch of diaries and getting the new interface in place so we can continue tagging. What I'm not sure about is how long that pause will be, but I should have a better idea once we've done more thinking about exactly what we'll need from the new interface, i.e. does it need to be the same as the current one, or can we simplify it in places?”

    I notice all the diaries are finished for tagging now – I think we will have just to wait and see what happens next and await information, hopefully an email will be sent to us. As Ral said also β€œMy plan was to ask Zooniverse to send out an email to all registered taggers”.

    Not sure whether Ral will have any more information for us at this time.

    I really miss tagging these diaires hope it continues one way or another :}

    Posted

  • stoat11 by stoat11

    All done for me too. It's been a pleasure to be involved and I hope that we might get a chance to continue at some point.

    All the best.

    Mike

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Hi Mike, best wishes to you as well and all others involved in this project.

    I did for a time tag the ANZACs on another Zooniverse Project for a while long time ago. I did not find it as interesting as this one we have all been transcribing for OWD. πŸ˜ƒ

    Posted

  • Storeman20 by Storeman20

    Just finished the last Diary, have thoroughly enjoyed doing them, and learned a lot along the way. I feel that there are a lot more stories from World War 1 to be told and passed on to the younger generations.

    Posted

  • erik.schaubroeckscarlet.be by erik.schaubroeckscarlet.be

    Finished https://www.youtube.com/watch?v=0LDArAJf7-c πŸ˜ƒ

    Posted

  • stoat11 by stoat11

    Brilliant πŸ˜ƒ

    Posted

  • ral104 by ral104 moderator, scientist

    Hi everybody,

    Sorry for the long silence - work is ongoing on the new interface. There have been some issues to overcome in terms of how it will work. To cut a long story short, I think we will have to accept random page allocation, which I'd initially been very resistant to as I know it will considerably change the experience of working through a diary chronologically. There is a way to keep that going, but it will have significant impacts on the data we can collect.

    I'm planning on getting the new workflows set up next week. These will have to be tested by a few people and go through Zooniverse's peer-review process before we can release the interface for general use, so unfortunately that's going to extend beyond the few remaining diaries we have here.

    But I just wanted to assure everybody that we are working on getting things up and running again. I think the best way to let you know once that's happened is by Zooniverse sending out a group email, so keep your eyes peeled on your inbox (or junk folder!)

    And THANK YOU for all that you've done on the current system - all your efforts are hugely appreciated and I hope you will all join us again very soon!

    Rob

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Ral, That is nice to know and thank you for the update. Also thank you for your on going work. Just another question, will this talk page still be active?

    Posted

  • erik.schaubroeckscarlet.be by erik.schaubroeckscarlet.be

    I fear that I'm not gone like that "random page allocation" thing (if it is what I think it is = pages not following?). Even now I found it annoying when some pages where missing.

    Posted

  • ral104 by ral104 moderator, scientist in response to marie.eklidvirginmedia.com's comment.

    It should be until the end of April, Marie. After that Zooniverse will be retiring old sites like this. So I'm hoping to have the new site up and running before then.

    Rob

    Posted

  • ral104 by ral104 moderator, scientist in response to erik.schaubroeckscarlet.be's comment.

    I know, Erik - it's definitely not my preferred way of doing it. The problem with continuing to work through the diaries the way we do now, is that the new system essentially would then treat an entire diary as one page. We wouldn't be able to classify the page type for individual pages as we do now, which in turn means we wouldn't be able to specify the data we collect depending on the page type. I'm still not 100% decided on this, but I think it's important we still retain the ability to distinguish page types, unless we accept that our old data and the new data we collect won't always be directly comparable.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Thanks for you reply Ral. I just wondered how I could return to the new site, if I missed out on the email that you mentioned would be sent.

    Seems the new set up may be complicated and not what we are used to and I suppose it will be different if random pages come through to tag. Still, rather random pages than no pages at all after all your hard work πŸ˜ƒ

    Posted

  • ral104 by ral104 moderator, scientist

    Thanks, Marie. You can also look on Zooniverse's project website. This is the history page: https://www.zooniverse.org/projects?discipline=history&page=1&status=live

    There may be some other interesting things there.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Thanks for your message Ral, I have looked at the Zooniverse site for other projects to do while awaiting the new inter face for the war diaries. I started transcribing the American Soldiers on the Zooniverse history project - Entries quite interesting and amusing and otherwise, from forms they filled in.

    Posted

  • Telegraphist by Telegraphist

    Hi, I have just signed up and completed the tutorial. However, I am unable to open a diary. Is this because the project has been put on hold for the moment.

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Yes, the project has been put on hold for the moment. As Ral, Moderator says, work is ongoing on the new interface. His messages regarding this are above.

    Posted

  • Telegraphist by Telegraphist in response to marie.eklidvirginmedia.com's comment.

    Many thanks Marie for your quick response and confirmation.

    Terry

    Posted

  • ral104 by ral104 moderator, scientist in response to Telegraphist's comment.

    Hi Terry,

    Thanks for joining us! As Marie says, we're on a temporary hiatus, but hope to be up and running again in the next couple of months. There should be a Zooniverse communique once our new interface is live.

    Rob

    Posted

  • clevelandtim by clevelandtim

    Oh No!, I was just reading through this in preparation for my students working with OWD in May. Each spring the last 5 years my advanced placement European history students have enjoyed working with the documents for a few days. Sorry to see this is gone.

    Posted

  • ral104 by ral104 moderator, scientist in response to clevelandtim's comment.

    Not gone, just on hiatus! We might be back up and running in time for your students. Keep an eye out for Zooniverse emails...

    Posted

  • 141Dial34 by 141Dial34 in response to ral104's comment.

    Excellent!!

    Posted

  • deehar by deehar in response to ral104's comment.

    Now that 6 months have passed I am assuming that this project is dead. Or is @ral104 still beavering away in some corner of a foreign field.....

    Posted

  • marie.eklidvirginmedia.com by marie.eklidvirginmedia.com

    Deehar, noticed your message re the War Diaries - I keep checking the talk page and I am wondering also whether the war diaries will return.

    Posted