Please make your dissertation welcoming to those not in your specialty and to those who do not share your language fully.
I often read dissertations when I am trying to get to know a new subject. I know many people read my dissertation, as it picked up lots of citations over the past few decades. I have no idea why people say only a few people will read your dissertation. That seems insulting. I have read a few undergraduate dissertations to find some formulas spelled out. More commonly, I read masters dissertations, and many more PhD dissertations, mostly those in fields that were not where I got my graduate training.
The exception is if you are in a harsh field where being nice to your readers is seen as a sign of weakness. Even then, you might be brave and buck the trend. Most importantly, do listen to your advisor. Don't spend forever writing this. Often the longer version takes no more time as you need not decide what to cut out, but at some point you need to pick a style and power through. Your committee can tell you to add detail or take out detail.
My advisor taught me that a good mathematician writes a "full-proof" version of every paper that nails down every detail, and then edits that down to make the published version. He wanted my dissertation to be the full-proof version of a few papers. Later, one is to only publish the short and efficient versions, but had the full version on a shelf to pull down if someone questioned you on a point.
If your dissertation is structured correctly, your committee can skip over sections that give background that they know well. Your advisor will get stuck reading it all. One does get paid for this, however.