This may not be the best solution, but there are lots of string alignment software packages readily available. Specifically, biological sequence alignment software is highly abundant. I actually did this once before using EMBOSS.
For many of these you'll need to convert your text to FASTA format, which I've written a script for:
#!/usr/bin/python2.7
import string
from sys import *
from Bio import SeqIO
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
def stripPunct(astr):
#For now, only output letters to satisfy EMBOSS
tmpstr = astr.translate(None, string.digits + string.punctuation + string.whitespace)
#tmpstr = astr.translate(None, string.whitespace)
return tmpstr
def main():
txtfile1 = open(argv[1],"r")
txtfile2 = open(argv[2],"r")
txt1 = txtfile1.read()
txt2 = txtfile2.read()
txtfile1.close()
txtfile2.close()
txtstrip1 = stripPunct(txt1)
txtstrip2 = stripPunct(txt2)
seq1 = Seq(txtstrip1)
seq2 = Seq(txtstrip2)
rec1 = SeqRecord(seq1,id="FILE_ONE")
rec2 = SeqRecord(seq2,id="FILE_TWO")
out1 = open(".tmp1.fasta", "w")
out2 = open(".tmp2.fasta", "w")
SeqIO.write(rec1, out1, "fasta")
SeqIO.write(rec2, out2, "fasta")
if __name__ == "__main__":
main()
Here is what a matching alignment looks like:
My use case was slightly different, and I was comparing between two different but related manuscripts we'd written. For your purpose, you'll want to find a multiple sequence alignment package, since you'll want to find matches other than the best match (which would just be the entire document since you are comparing a document to itself).
diff
on a plain-text file. What software do you use to write? Perhaps the program you are using to write has a more elegant solution built-in.diff
is really only suitable for comparing different versions of the same document, not searching for similarities within the same document. I think the best solution to your problem is a good, thorough dose of proof reading :). Remember: there are only two kinds of thesis. A perfect thesis, and a finished thesis. Don't sweat the tiny details too much. Best of luck!