Sorry if I have overlooked this but are the dice rolls capped at 5? In other words, if a dude with a rating of 5 in an attribute gets a +1 modifier do you ignore any value over 5 when rolling dice?
No, it's not capped. Rules Reference p. 18:
2. Roll Dice: He rolls a number of dice equal to his dice pool.
• If his dice pool is less than one, he rolls one die instead.
• If his dice pool is greater than the number of dice available, he rolls as many dice as possible, records the results, and rolls the dice again until he has rolled dice equal to his dice pool.
BONUS QUESTION! I saw a random mythos thing come up that called for 2 horror and 1 damage with a roll that could reduce the effect but I was unclear how that worked. If a player rolls less than 3 do they choose which effects (if any) to reduce or is there a sequence?
Yes. Rules Refence p. 13:
A test declaration that includes the word “negates” following the skill icon indicates that the test prevents Damage or Horror.
• For each success result rolled, the investigator prevents one Damage or one Horror, his choice.