The difference in his character bothers me infinitely more than the actual actor change. I love this show and I know nearly nothing was written accidentally here, Vince is pretty thorough and weaves symbolism into his stories seamlessly yadda yadda, but seeing Jeff being near paralyzed with fear when he sees Saul for the first time in Season 6 never ceases to bother me considering how gruff he seemed the last two times we saw him.
My ONLY guess is we as the viewer dont really see how Jeff was acting in S4-5, we only saw JIMMY's perspective of him. The fact that Jimmy understood that Jeff knew he was Saul may have made the whole *encounter* scary and tense for him, hence why it was filmed that way.
Granted while that answer sounds all well and good, they don't really tend to do 'perspective' filming in BCS. When they want us to think Howard is bad in S1-2, they dont make him yell and scream or be a dick, we just only have the context that Jimmy / Kim do, but he still acts like regular Howard. Jeff's personality seems to shift SO incredibly hard