Ex scared to see me

He's shown you who he is. It's probably not about you, but he clearly has mental health problems that impact his ability to be a good partner to someone or to follow through. Do you want that kind of partner who you can't trust or rely on for any support in tough times? Do you want to be in a "care giver" role that could lead straight to codependency, as these are his personal issues he needs to get probably professional help for to learn how to manage, and you are not his therapist and can't and shouldn't be in that role anyway as a mature romantic partner?

If you just "get back together" without him working on resolving his issues, nothing changes, you get more of the same frustration and push/pull no matter if he means well or not. For relationships to work and a foundation of trust to be established, words and actions and feelings need to be consistent. If someone says one thing and does another, then the lack of consistency speaks loudest, so do not look for hope in the inconsistency. Listen to whichever is telling you the more "negative" story in that case, whether that's the words or the actions. Not because you should think negatively about people or assume the worst, but in these situations where someone has shown you they are inconsistent over and over, that's where the truth lies about their capacity for a relationship.