don't blame you. he may just go back in his shell if he sees a hand come close to his face. that's what mine do. however, lemi (my most active) has allowed me to (after thoroughly washing my hands of course) to get a clump of sand off of his eye. he stayed out the whole time. he cleaned his eye when i put him down. it was kinda interesting that he trusted me enough to do that.
what's interesting is how they get the water out of the sponge. we have to squeeze it, they can't squeeze, so do they just stick their little claw into one of the holes and get it moist or something?
i have always thought the sponges were really just used for bumping up the humidity and for crabs to hold onto. I am sure they can get some water out, but it seems easier for them just to drink out of the bowl, mine do anyways. i don't really know!