Dev,
Split out the thick outer portion (for storage and transportation purposes this portion is retained by mechants). Then cut the narrow pith in cross-section. Each time you cut you will get fibres coming out. Remove the fibers. Another method is after chopping into small pieces (Always put chopped pieces in water, else it will become discoloured very quickly), use a long needle (the one used for frying pappadum) or chop-sticks and stir the water along with the pieces. This will collect all the extra fibres out. Be careful never to touch your cloths with banana-stem. No detergent promises you remove the stain!