Why are there flake like stuff inside my ear?

Recently I went for a medical review had a hearing test. One of my ear could only hear sounds of varied frequency at consistent,lower dBs than the other ear. The medic said I have ear wax as i dig my ears. He gave me olive oil to put inside my ear once a day and ask me to return for a follow up treatment and retest. So I followed his instructions and use the oil and abstained from digging my ears. But today I went to touch it today and found these dried up stuff that are stucked to my ears at the ear canal where your pinky can reach on both ears. ( i know I am not suppose to do that I think?) is that normal? Or should I remove it? Erm, I stopped cleaning both my ears last weds, 22/6/2011 and today is Sunday 26/6/2011. I was wondering if anyone can give me advice on what to do? Thank you!