Gambling applies to after when a player defeats a boss. A player can pay any amount of col, depending on the minimum and maximum amount that is able to be payed. The minimum and maximum amount of col that can be payed for gambling, is decided upon the boss's combat level and what floor you are currently on. Gambling by paying a higher amount of col, you will gain more experience into your gambling skill than paying a smaller amount of col.
Once paying to gamble your boss loot reward, you will re-roll the loot for yourself (Not your party). There is a probability chance of receiving something better or worse than what you would have gotten. This depends on the amount of col that you payed, a higher amount of col payed has a higher chance of you receiving something better.
You can gamble a boss's loot reward in the rewards window after defeating a boss. The option to gamble your rewards will be disabled a minute after it is available, so you only have a small amount of time to look over what you have won then decide if you'd like to re-roll.
Having a high level skill in gambling, and paying a high amount of col both at the same time will give you a very high chance of getting a rare boss drop for the current floor and level.