I don't know the technical term but the when you define the function and you put "..." at the end, it essentially means "put any extra parameters passed into this function into ...", this is very useful for a function that handles events because very often you will be dealing with multiple events that return different number of parameters. So when CHAT_MSG_SYSTEM triggers, ... contains all the variables the event lists on the wowprogramming page, in order.
In this case we're only interested in the first value "message" because the author of the message can be parsed from the message and all the other values we don't care about.
This line basically says "set message equal to the first value in ...", if we cared about the rest of the values we could instead do
local message = ...
and then we would have those values in variables as well.
local message, sender, language, channelString = ...
In this case since we're only using that function defined in SetScript() to handle one event, we could've easily just defined the parameters as
but it's good practice to use ... there because if you later decide you need to register another event to the frame, you can handle the arguments of that event as well without having to modify your code. That's also the same reason I added the 'if event == "CHAT_MSG_SYSTEM" then end' structure even though with only one event it's completely superfluous.
function(self, event, message)
I don't see why rollResult would equal nil, are you sure you got the variable's name correct? I use that exact matching pattern in one of my addons and it's always worked for me.