My Aunt worked on systems for explaining early generation networks for medical diagnoses: http://link.springer.com/article/10.1007/BF01413743