5

I am reaching the final stage of my dissertation which means editing, rewriting, and more editing. I am looking for a tool which can help me in this process by checking for similarities within a single text of 150-200 pages. I will probably detect similar phrasings of key points manually, but I could use some automated help to make sure I don't repeat things in the same words. I know Word and other software can compare documents to look for similarities between two documents, but is there a way to look for similarities within a single document?

Thanks.

5
  • 1
    On OS X or Linux you can use diff on a plain-text file. What software do you use to write? Perhaps the program you are using to write has a more elegant solution built-in.
    – Moriarty
    May 29, 2014 at 20:24
  • A quick and dirty tool that you can make is a suffix tree for a text-only file. Essentially, let the jth entry of an array be (a truncation of) the string starting at the jth byte of your text. Sort the array, and find out which entries occur most often, or which prefixes of entries occur most. This will give you an idea of what actually is being repeated, and can indicate cases where the same root word is being used often. I recommend awk to build such a tool. May 29, 2014 at 20:30
  • @Moriarty I have settled on using Microsoft Word because of its excellent integration with Zotero and because it's the standard software used in my department (thus making the process of compatibility easier, which was definitely a problem when I used Open Office before). I know that Word can compare two separate documents, but I am not aware of a way to do something similar in 1 document. Could you maybe advise me how to use the diff command in OS X in the terminal? I have no idea ;)
    – silvertje
    May 30, 2014 at 18:15
  • 1
    @silvertje I fear I misread the question - diff is really only suitable for comparing different versions of the same document, not searching for similarities within the same document. I think the best solution to your problem is a good, thorough dose of proof reading :). Remember: there are only two kinds of thesis. A perfect thesis, and a finished thesis. Don't sweat the tiny details too much. Best of luck!
    – Moriarty
    May 30, 2014 at 18:25
  • @Moriarty Thank you. I will go for the finished thesis and hope I will catch slight overlaps during the proof reading process ;)
    – silvertje
    May 30, 2014 at 18:49

1 Answer 1

1

This may not be the best solution, but there are lots of string alignment software packages readily available. Specifically, biological sequence alignment software is highly abundant. I actually did this once before using EMBOSS.

For many of these you'll need to convert your text to FASTA format, which I've written a script for:

#!/usr/bin/python2.7
import string
from sys import *
from Bio import SeqIO
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord

def stripPunct(astr):
    #For now, only output letters to satisfy EMBOSS
    tmpstr = astr.translate(None, string.digits + string.punctuation + string.whitespace)
    #tmpstr = astr.translate(None, string.whitespace)
    return tmpstr

def main():
    txtfile1 = open(argv[1],"r")
    txtfile2 = open(argv[2],"r")
    txt1 = txtfile1.read()
    txt2 = txtfile2.read()
    txtfile1.close()
    txtfile2.close()

    txtstrip1 = stripPunct(txt1)
    txtstrip2 = stripPunct(txt2)
    seq1 = Seq(txtstrip1)
    seq2 = Seq(txtstrip2)
    rec1 = SeqRecord(seq1,id="FILE_ONE")
    rec2 = SeqRecord(seq2,id="FILE_TWO")
    out1 = open(".tmp1.fasta", "w")
    out2 = open(".tmp2.fasta", "w")
    SeqIO.write(rec1, out1, "fasta")
    SeqIO.write(rec2, out2, "fasta")




if __name__ == "__main__":
    main()

Here is what a matching alignment looks like: enter image description here

My use case was slightly different, and I was comparing between two different but related manuscripts we'd written. For your purpose, you'll want to find a multiple sequence alignment package, since you'll want to find matches other than the best match (which would just be the entire document since you are comparing a document to itself).

1
  • Thanks for these suggestions. Unfortunately, I think they fall outside of my technical abilities to actually run or use one of these programs/scripts.
    – silvertje
    May 30, 2014 at 18:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .