I feel that historical novelists owe it to our readers to try to be as historically accurate as we can with the known facts. Obviously, we have to fill in the blanks. And then in the final analysis, we're drawing upon our own imaginations. But I think that readers need to be able to trust an author.
In writing my historical novels, I have to rely upon my imagination to a great extent. I think of it as 'filling in the blanks.' Medieval chroniclers could be callously indifferent to the needs of future novelists. But I think there is a great difference between filling in the blanks and distorting known facts.