Drawing and sketching is a very fun and relaxing activity. I need to get back into it, but I always used to listen to music and let my creativity flow, and other optimistic and rainbowy statements.
I relate to you in that it's a bit of a personal activity for me, and I enjoy privacy whenever I sketch. If your peers/observers/Big Brother-esque dictators can't understand or respect your privacy, then you should not blame yourself, but at the same time they should understand that it's simply something you like to do. I don't fully understand how the materialism bit plays into this, but if they think you're wasting time or something of similar effect, that's quite a shame since everyone needs their own personal time or activity; not everything is about generating money.
Not that you need to tell me, but the irony of this whole thing is now I'm curious as to what you're sketching. You don't need to tell me, of course, but I'm just wondering if you're doing everyday doodles or
super-bizarre-disturbing works.
'Cause stuff like that image WILL make people stare if they observe you drawing it.